Open all links on a cfm page

[weezerboy]

I have a cfm page with 30-40 links on it. These links send emails with attachments to different clients

I need to un this page and all of these links via a scheduled event

Rather than creating 30-40 scheduled events to open each link and send the emails, is there a way to go to the page and have CF open all the links?

Link1
Link2
Link3
Link4
.
.
.

.

Link40

[EddieLotter]

Why bother "opening" the links? You have the logic that created the links. Use the same logic to send the e-mails in your scheduled task.

[weezerboy]

Its not just sending emails but a lot of other logic, like creating PDFS etc - seems like too much

[EddieLotter]

Then you need to encapsulate the logic so that there is no duplication and it can be called from a link or from a scheduled task.

[WolfShade]

It _CAN_ be done, but as Eddie has pointed out, it's not the best approach.

Create a template, don't bother with UX/UI since no user will actually see it (you can even set it so that if the user-agent does not include CFSCHEDULER, it won't run), put all the logic in place to generate PDFs, etc., and set it up in Scheduled Tasks.  BAM, done.

V/r,

^ _ ^

[Charlie Arehart]

And though others are rightly focused on other solutions, it's worth noting that on the surface, no, there is no feature in CF that would read a page and run the links on that page.

That concept is typical of "spiders" (or crawlers), such as are used by search engines (to trawl through all pages on a site, which then find and follow links in each page, saving all the content and links to run again the next day, to keep track of content on sites that are to be searched).

There's no feature in CF to spider a site, let alone a page. Some may want to note that since CF includes Solr, which is based on Lucene, there is indeed a lucene-based crawler called Nutch (http://nutch.apache.org​​). But there is no specific integration of CF with that.

And while I'm not aware of any other sort of feature or tool within CF that could easily do the sort of scanning of a page to find links that you seek, I will note that there is a a free online tool that does purport to find links on a page and return them as a simple text list. Try this link:

https://api.hackertarget.com/pagelinks/?q=learncfinaweek.com

And you could call that with CFHTTP then loop over the result doing other cfhttp calls to get whatever they offer. Note that you are limited to running 100 such requests to the api per day (unless you pay for more).

That said, it seems to be looking for only certain kinds of links (like a href references) and not others that would seem to us to be links (perhaps javascript-based), because if you point the tool to the Adobe forums (which has many links, of course), it returns none:

https://api.hackertarget.com/pagelinks/?q=forums.adobe.com

There are still some other things about the tool (it doesn't follow redirects, etc.) that you can read about here:

https://hackertarget.com/extract-links/ 

Hope that's of some help, to you or other readers who may find this discussion.

[BKBK]

weezerboy  wrote

Rather than creating 30-40 scheduled events to open each link and send the emails, is there a way to go to the page and have CF open all the links?

Link1
Link2
Link3
Link4
.
.
.

.

Link40

Sure thing. Use

<cfhttp url="linkX" >

to open each link

[WolfShade]

Yeah, you _could_ use CFHTTP to grab the contents of a local .cfm page and parse it for links.. but, WHY?  This could be also done in AJaX with zero cost to the webserver or the CF server.  Let the client do the work.

But, again, WHY?  Create a page with no concern about UX/UI, put it in Scheduled Tasks, done.

Just my two cents.

V/r,

^ _ ^

[Charlie Arehart]

Wolf, you ask why one would go the other route and not edit the page creating the links to be more dynamic and then use a variant of the page without ui to call the links.

Only Weezer can tell us what's possiblr for him, but since I offered on of the alternatives (and therefore your question may be to me as well), I'll just propose that someone in his shoes may not be free to edit the page in question.

And one may assert then that he just extract the url's into a template (or db) to then call them WITHOUT scanning the page that offers them.. But it is possible that the page and links change.

So I simply offered the option if either may be the case. As I said at the open of my first comment, you and others are indeed right to press if there may be a better way.

I just offered an option if those don't work for him, and in case someone else reading this in the future may need it, perhaps for a page of links that may even not be on their server, or not a CF page they control, etc.

[BKBK]

WolfShade  wrote

Yeah, you _could_ use CFHTTP ... but, WHY? 

Because of these requirements:

• open each link
• send the emails

I am assuming that weezerboy​ wishes that ColdFusion perform these tasks.

[weezerboy]

I appreciate all of the input on this.

Here is why I need to open the links individually as opposed to encapsulating the 30-40 pages script into 1 page.

Each of the 40 pages has about 500+ lines of code that create PDF, do data merging and then sends large PDFs.
Lots of server intensive stuff

Some of the pages take 30-45 seconds to complete.

So I really need to keep these separate and not put them into 1 page.

I think I am going to look into something like this  ...<cfhttp url

[WolfShade]

If each page is not dependent upon a previous page being processed before it starts, you can CFINCLUDE each into a CFTHREAD and process them that way, on a single page.

HTH,

^ _ ^

[weezerboy]

can you give an example of this Wolf?

[Dave Watts]

I would probably be a bit wary of doing this. CFTHREAD is a great way to let a bunch of tasks run concurrently, but ... you end up running a bunch of tasks concurrently, which is not always such a great idea! It's good when you want to minimize the amount of time a user has to wait, but it takes resources away from other tasks. In this case, it sounds like there's no interactive user, so it might be better just to have this run for a long time without using a lot of resources at one time.

Dave Watts, Fig Leaf Software

[Dave Watts]

I think everyone here is a little bit confused about exactly what you're trying to do, including me. Maybe it's just me actually, I dunno. But in general, you're better off not using CFHTTP from your server to invoke another page on your server, because it's very inefficient - it involves using one CF request to run another. And if your original requests run slowly by themselves, that means you now have two slow-running requests for each.

So, what most everyone here is trying to do is figure out a way you can avoid this, or minimize the impact of this.

Do you have the links stored in a database? If so, you could write a program that queries the database to get the information that you need and performs whatever logic would normally be performed by a user visiting the links manually, without involving additional HTTP requests. That would be ideal. Your program could process these one at a time, in whatever fashion you liked. It could run serially: process the first one, put itself to sleep for a while, process the second one, etc.

If for whatever reason you don't have the links stored in a database and the only way to retrieve them is by reading files (for example, the links are static text in your primary file), perhaps you could read them via the filesystem (CFINCLUDE, for example) instead of HTTP and extract the links. That would move the process of reading them from your web server, and that would be more efficient.

If for whatever reason you can't do that, you could use alternatives to process them via HTTP, instead of using CF to do this. Curl or wget are the traditional command-line programs for this. (You really don't want to start figuring out Apache Nutch for something like this, we'll never hear back from you again!)

So, I guess your best bet is to describe the whole thing in as much detail as you can, and you will get more refined solutions from us.

Dave Watts, Fig Leaf Software

[BKBK]

I agree with Dave. A 30 to 45 minute request is a fair price to pay for the transaction.