Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 5 of 5
  1. #1
    New to the CF scene
    Join Date
    Jul 2018
    Posts
    2
    Thanks
    1
    Thanked 0 Times in 0 Posts

    is this possible?

    Hi Everyone,

    I have almost 0 coding experience. I am trying to simply find out if this type if thing is possible. From what I have heard from other code proficient folks, ANYTHING is possible.

    The situation: I am a part time editor of scientific papers. Each paper I edit I receive a small commission for. However, I am part of a larger group of editors. When there is a paper available for edit, an email is sent to the larger group with a link that if clicked will claim the paper to be edited. The paper never stays available for very long due to the large size of the group and everybody wanting to claim them. So, I would like to write some code that will recognize when a paper is available to my group and claim it immediately on my behalf. I also want it to be easy to turn on and off in the case I dont want to claim papers at a particular time.

    So, is this possible? Where should I start? Thank you!

    dbradley77

  2. #2
    Supreme Master coder! Philip M's Avatar
    Join Date
    Jun 2002
    Location
    London, England
    Posts
    19,544
    Thanks
    221
    Thanked 2,712 Times in 2,686 Posts
    This may be possible, but not using Javascript, which is purely a client-side language. Ask a mod to move this thread to a more appropriate forum.

    All the code given in this post has been tested and is intended to address the question asked.
    Unless stated otherwise it is not just a demonstration.

  3. Users who have thanked Philip M for this post:

    dbradley77 (Jul 17th, 2018)

  4. #3
    New to the CF scene
    Join Date
    Jul 2018
    Posts
    2
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Thanks so much for the input. How would I ask a moderator to move the thread?

  5. #4
    Super Moderator vinyl-junkie's Avatar
    Join Date
    Jun 2003
    Posts
    3,437
    Thanks
    3
    Thanked 50 Times in 50 Posts
    Quote Originally Posted by dbradley77 View Post
    How would I ask a moderator to move the thread?
    Click on the Report Post link at the bottom left of every post (triangle icon with an exclamation point). What you SHOULDN'T do is re-post your request in a new thread.

    I deleted your new thread and moved this one.
    Music Around The World - Collecting tips, trade
    and want lists, album reviews, & more
    SNAP to it!

  6. #5
    New to the CF scene
    Join Date
    Jul 2018
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Thumbs up It is very possible!

    Quote Originally Posted by Philip M View Post
    This may be possible, but not using Javascript, which is purely a client-side language.
    This is wrong. Javascript has been both client and server language for many years thanks to tech such as Google's V8 and Node.js. https://en.wikipedia.org/wiki/List_o...mplementations


    Quote Originally Posted by dbradley77 View Post
    So, is this possible? Where should I start?

    This is very possible! I have done many a project such as this, especially in Javascript. I created a quick and dirty example to demonstrate. This program is meant to run on a timer, such as cron (linux) or Windows Task Scheduler. Every time the program runs, it downloads a list of articles from (example website) hackernews, and essentially clicks every article on the page that is new. For the example I used article links, but in your case they would be buttons or whatever you use to claim a paper. All this is done in a terminal window, without a web browser.


    Code:
    // I want to use some libraries to keep code nice and short.
    
    // Here I am requiring (importing) a library called cheerio to help with HTML scraping and manipulation.
    const cheerio = require('cheerio');
    
    
    // Here I am requiring superagent, a library for making web requests.
    const request = require('superagent');
    
    
    // Here I am requiring fs (filesystem), which will act as our program's long term memory.
    // We need this (or a similar method) to save our progress and make sure we don't claim articles that we already claimed
    const fs = require('fs');
    
    
    // Here I am requiring path. This is optional but it will help us build file paths such as
    // C:\Users\Chris\Desktop or /home/chris/Desktop
    // without them turning out all funky such as C:\\Users\\ChrisDesktop
    const path = require('path');
    
    
    // Here I am requiring os (operating system.) This will help our program
    // know where to save things, regardless of what Operating System we may be running.
    const os = require('os');
    
    
    
    
    // Our dependencies are all taken care of. Let's take care of some configuration!
    
    // Let's configure where we want to save our long-term-memory.
    // We'll put it in a folder on the desktop.
    const saveDirectory = path.join(
      os.homedir(),
      'Desktop',
      'paper-detector',
      'memory.txt'
    );
    // what we did there is simply generate a {string} file path:
    // 'C:\Users\Chris\Desktop\paper-detector\memory.txt'
    // we will use this file path later.
    
    // now we will create the file and it's parent directory
    // if they don't already exist
    const parentDirectory = path.resolve(saveDirectory, '..');
    if (!fs.existsSync(parentDirectory)) {
      fs.mkdirSync(parentDirectory); // create parent directory
    }
    fs.openSync(saveDirectory, 'a');
    
    
    
    
    
    // Here I use superagent to GET the content of hackernews. I'm using Hackernews as an example,
    // but dbradley77 would use something like https://scientificpapers.example.com/paperfeed/
    request
      .get('https://news.ycombinator.com/')
      .end((err, res) => {
    
        // The request has been made at this point.
        // Now we need to handle any errors that may have occured while getting the news feed.
        if (err) {
    
          // This is VERY simple error handling.
          // Since this program is meant to run on a timer,
          // a better error handling would notify the human by sending an e-mail or something.
          throw err;
    
    
        }
    
    
        // the following code runs only if there was NOT an error.
        // we will call a {function} scrapeArticles(), which uses cheerio
        // to parse the HTML. We pass the {string} res.text variable to scrapeArticles
        // the res.text variable contains the response from hackernews, in our case
        // the response is just HTML.
    
    
        scrapeArticles(res.text, function(err, articles) {
          if (err) throw err; // handle errors in scraper. very rudimentary.
    
          // after the article links are found, time to click the links!
          clickArticles(articles, function(err) {
            if (err) throw err;
          });
        });
    
      });
    
    
    
    
    
    function scrapeArticles(text, callback) {
    
    
      // In this example, we will imagine each article on hackernews is an article on scientificpapers.example.com.
      // Whatever the action to claim an article may be, we can perform that action using Javascript.
      // We will keep things simple in this example and assume that clicking an article is all it takes to claim the article.
    
      // Ok let's scrape some HTML and get a list of articles.
      // cheerio parses all the HTML we feed it.
      const $ = cheerio.load(text);
    
      // Now we can use the output of cheerio and get the data we want.
      // we can use CSS selectors with cheerio.
      // From my poking around in Firefox's Inspector, I learned that hackernews
      // articles are listed in a HTML <table>, each title existing in a <tr> with CSS
      // class "athing"
      // We use this information to build a CSS selector for cheerio.
      var itemList = $('tr.athing')
      console.log(itemList.text())
    
      // the titles are nice, but what we really want is the link, or the <a href...>
      var linkList = [];
      $('a.storylink')
        .toArray()
        .forEach(function(link) {
          linkList.push(link.attribs.href);
        });
    
    
      // now we have a list of links, so we can show them in the log
      console.log(linkList)
    
    
      return callback(null, linkList);
    
    }
    
    
    
    function clickArticles(articles, callback) {
    
        // now that we have our list of links, all we gotta do is click them all!
        // of course, we only want to click new links, so we memorize the links we click
        // and store them in our program's long-term-memory.
    
        articles.forEach(function(articleLink) {
    
          fs.readFile(saveDirectory, function(err, memory) {
            if (err) throw err;
            if (memory.indexOf(articleLink) == -1) {
              // only click the article if it doesn't exists in the memory.
              request
                .get(articleLink)
                .end(function(err) {
                  if (err) {
                    console.error(`Problem encountered while loading ${articleLink}. Skipping it for now.`)
                  } else {
                    fs.appendFile(saveDirectory, articleLink+'\n', function(err) {
                      if (err) throw err;
                    })
                  }
                })
            }
          })
        });
    
    
    
    }


 

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •