AE using socket to import picture from URL - Need Your Help

Report · Oct 31, 2012

I have a project that works with local network using the path name. I want to update the project to replace all placeholders with a picture located at a URL address instead of the local network. I have no experience with socket coding and can't seem to find the right snippets of code on the internet to get me through this point of using URL as a file location.

I tried this code (using a picture at this URL as an example);

webConnect= new Socket;

if(webConnect.open("http://distilleryimage3.s3.amazonaws.com")) {

webConnect.write('GET /2940b7e0236511e2af9022000a1f9a23_7.jpg HTTP/1.0nn');

response= webConnect.read();

webConnect.close();

alert('complete');

} else {

alert('fail');

}

Any help will be appreciated. I need the results of the socket connection to be a object I can use in the statement

curItem.replace(new File( <picture path> );

Thanks again.

Report · Oct 31, 2012

Just looked in the adobe javascript documentation and you should not put the http:// and you should put the http port number 80 at the end. So the connection should be "distilleryimage3.s3.amazonaws.com:80".

At the GET, make sure you have slashes before both "n".

Report · Oct 31, 2012

Thanks, that was most helpful. At least I'm getting a response from the server. Working on the idea of image buckets and such. Some trial and error and more reading I was able to get this socket response; This was response from

webConnect.write('GET /distilleryimage5/17bdfa64237411e2a23c22000a1f9d66_7.jpg HTTP/1.0\n\n');

How do I save the results into an object that curItem.replace(new File( <Socket response> ); will work with? I did change the GET to 'binary' insteaed of HTTP:/1.0\n\n but just don't know what to do with it next.

You help and patience with my questions has been greatly appreciated. Learning curves and patience are not easily to over come or use at time. Thanks again.

Report · Oct 31, 2012

Now you need to parse that response.

Here to get you started :

var response =  webConnect.readln();
var statusCode = new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/g).exec(q)[1];
if (parseInt(statusCode) != 200) webConnect.close();
else{
    response =  webConnect.readln();
    /*read all the necessary lines;*/
    webConnect.close();
}

I suppose there is an undefined number of line in that header. So you'll read lines by lines and look for content that you want. Eventually you'll need to know the content-length that you'll feed to the webConnect.read(contentLengthValue). You might do it using regexp like i did for the first line (http://gskinner.com/RegExr/). You should look at the definition of a http response for how to finish processing this header, but it looks like there is an empty line before feeding the data.

Report · Nov 01, 2012

Thanks again for your help. I'm inching towards a working bit of code. I'm still researching socket service protocol because I'm just typing in code and isntructions sometimes and just don't know why I'm doing it.

What is the \n\n after the HTTP1.0 in the GET command?

It does appear that the data (picture) is at the end of the initial response, I'm just not able to separate it from the header and create a valid image file.

Perhaps it is understanding what the \n\n means. Also do I need the\\n\n if I use BINARY instead of HTTP1.0 ?

Thanks again for all of you help. I feel as though I'm one step away from this working.

Report · Nov 01, 2012

Ok, to get the data at the end, it is like I said, you have to get the value of content-length and use that value for webConnect.read(content-lengthValue). From what I understand, the empty line means the end of the header, after that, it is the data.

To process the header, you'll do an while loop with a readln command that you'll parse. (right now, you should replace it by a for loop to prevent and infinite loop)

So something like (i didn't test it, so there is possiblity there is some errors):

var contentLength = ""

var statusCode = ""

for (var i = 0; i < 20; i++){

var response = webConnect.readln();

if (new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).test(response)){

statusCode = parseInt( new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).exec(response)[1] );

}

else if (new RegExp(/Content-Length: ([0-9]+)/).test(response)){

contentLength = parseInt( new RegExp(/Content-Length: ([0-9]+)/).exec(response)[1] );

}

else if (new RegExp(/^\s*$/).test(response)){

break;

}

if (statusCode != 200) webConnect.close()

else{

var data = webConnect.read(contentLength);

var file= new File(Folder.temp.absoluteURI + "/httpSocket.jpeg");

file.encoding='BINARY'

file.open ("w");

file.write(data);

file.close ();

file.execute();

}

If not, I don't know the thing wbout BINARY instead of HTTP.... for the \n\n, it is just empty line to tell the server to stop reading. And in fact, it should be \r\n from what i saw on the web - cause of the line carriage and line return diffrences.

Report · Nov 01, 2012

OK, this is my working code, that appears to fetch the right size of data after the text (the pic is 121KB) but it is not saving a valid jpg file. What appears to be a blank line between the text and the data is something so I changed the last else to recognize the last text line. The new RegExp(/^\s*$/).test(response) was not breaking and it was reading too many lines


	webConnect	= new Socket;

if(webConnect.open( "distilleryimage5.s3.amazonaws.com:80")) {

webConnect.write('GET /distilleryimage5/17bdfa64237411e2a23c22000a1f9d66_7.jpg HTTP/1.0\n\n');

var file= new File("httpSocket.jpeg");

file.encoding='BINARY'

file.open ("w");

var contentLength = ""

var statusCode = ""

for (var i = 0; i < 20; i++){

var response = webConnect.readln();

if (new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).test(response)){

statusCode = parseInt( new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).exec(response)[1]);

}

else if (new RegExp(/Content-Length: ([0-9]+)/).test(response) ){

contentLength = parseInt( new RegExp(/Content-Length: ([0-9]+)/).exec(response)[1]);

}

else if (new RegExp(/Server: AmazonS3/).test(response) ) {

break;

}

if (statusCode != 200) webConnect.close()

else{

alert('statusCode=' + statusCode + ' contentLength=' + contentLength);

var data = webConnect.read(contentLength);

alert(data.length);

file.write(data);

}

file.close();

file.execute();

}

Report · Nov 01, 2012

OK, this comes close to working. I can read the jpg file with QuickTime, though it is corrupted and the image is different color or hue. Nothing else will read the jpg file, it so my efforts to use After Effects have still not been met.

I have not idea why I have to read 1 byte at the beginning of the data set and discard it. I can't find enough information about the jpg header format but it seems there is corruption at the top of the downloaded file. Since I'm downloading the 'contentLength', it's odd that midstream I get corrupted data. It appears to be consistent too.

Anyway, I am about at a lose.


	webConnect	= new Socket;

	if(webConnect.open( "distilleryimage5.s3.amazonaws.com:80","binary")) {
		webConnect.write('GET /distilleryimage5/17bdfa64237411e2a23c22000a1f9d66_7.jpg HTTP/1.0\n\n');

var file= new File("httpSocket.jpg");

file.encoding='BINARY'

file.open ("w","binary");

var contentLength = ""

var statusCode = ""

for (var i = 0; i < 20; i++){

var response = webConnect.readln();

if (new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).test(response)){

statusCode = parseInt( new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).exec(response)[1]);

}

else if (new RegExp(/Content-Length: ([0-9]+)/).test(response) ){

contentLength = parseInt( new RegExp(/Content-Length: ([0-9]+)/).exec(response)[1]);

}

else if (new RegExp(/Server: AmazonS3/).test(response) ) {

break;

}

if (statusCode != 200) webConnect.close()

else{

alert('statusCode=' + statusCode + ' contentLength=' + contentLength);

var data = webConnect.read(1);

		var data = webConnect.read(contentLength);
		file.write(data);

}

file.close();

file.execute();

}

Report · Nov 01, 2012

The extra byte is cause you changed the regexp from /^\s*$/ to /Server: AmazonS3/. Put it back to /^\s*$/

I have the reading part a little bit better but it is always shorter doesnt' understand why. At least, the script stops when/if the server close the connection.

var data = "";

while (data.length <= contentLength){

data = data + webConnect.read(contentLength);

if (!webConnect.connected) break;

}

webConnect.close()

delete webConnect;

I also found this page that is really good explained. From what I understand, it could be that the GET should be implemented for HTTP/1.1 and maybe adding a keep-alive connection arguments.

http://www3.ntu.edu.sg/home/ehchua/programming/webprogramming/http_basics.html

Report · Feb 07, 2018

I have found a solution to this for anyone who may be interested:

It's fairly simple, instead of using the socket object, what I've done is use wget, which is a command prompt tool that can download files via command prompt, since you can execute commands through extendscript with system.callSystem, you can execute the wget tool to download the file from the url, all you need to do is download the wget tool from here: GNU Wget 1.19.4 for Windows then place wget.exe into the folder you want to download images to

I've written a snippet that works with After Effects that will download the image with wget, then import that file into After Effects, just change the paths to suit:

var folderLocation = "d:/wget";

var imageURL = "https://i.imgur.com/pKY6r1K.jpg";

var myCommand = "cd /d " + folderLocation + " & wget " + imageURL;

system.callSystem("cmd /c \"" + myCommand + "\"");

var path = new File(folderLocation + "/" + (imageURL.split("/")[imageURL.split("/").length - 1]));

if (path.exists) {

var io = new ImportOptions(path);

var videoFile = app.project.importFile(io);

} else {

alert("There was a problem importing the image");

}

Report · Mar 21, 2018

stevelewisza – thanks for posting! This brings me a little closer to a solution to a variation on this technique I've been trying to wrap my head around, which is pulling items from an Adobe Shared Library into AE. I think because the libraries are cloud based, each asset should have a web address, which could be called using your snippet. (Since I can't figure out a way to access Adobe Libraries directly inside of AE).

Many of the end users of my projects will use Macs, however. Do you know of a similar software that would be compatible with Mac? I suppose it wouldn't be too difficult to version the script out for each OS. If there is something similar, would the process of calling the .dng via system.callSystem work the same way it does to call an .exe in Windows?

Any insight would be greatly appreciated!

Report · Mar 29, 2018

curl is probably your best bet on macs, (wget is available, but curl is built-in).