Skip to main content
S
s_filipowicz
Known Participant
August 22, 2011
Question

How to transfer large amount of data between websites

  • August 22, 2011
  • 3 replies
  • 1539 views

Hi everyone,

I have a real estate website which has about 7000 properties. This data I want to share and transfer to another website (also CF).

I made a nice remote cfc which gives XML which can be called with a <cfhttp url=".." >. Creating the XML takes pretty long 10 minutes, and then parsing the XML takes much longer, 1 hour or longer. Not really ideal.

Does anyone have any suggestions on how I can better do this??  The idea is that this transfer will be done every few days.

Thanks!

    This topic has been closed for replies.

    3 replies

    J
    JR__Bob__Dobbs
    Inspiring
    August 22, 2011

    Rather than processing the data using CF I would investigate bulk import/export tools supported by your database server.  MS SQL Server, MySQL, and others can import/export data to delmited text files.  These import/export tools will *probably* perform faster than using CF's XML tools to process thousands of records.

    A few questions:

    1. Do you control both sites?

    2. What database server are you using?

    3. Can you post a sample of the data?

    S
    s_filipowiczAuthor
    Known Participant
    August 22, 2011

    Hi,

    Thanks for the replies everyone!

    1: Yes I control both sites.

    One website is on a shared hosting where I can't install software, the other website is running on a dedicated server to which I have full access to install what I want.

    2: Using MySQL on both

    3:

    <property COUNTRY="Bahrain" ID="906140141208">
    <home_type>7</home_type>
    <home_type_text>Compound</home_type_text>
    <home_area>14</home_area>
    <home_area_text>Saar</home_area_text>
    <photos>
    <photo1>906140141208_BAHRAIN_PROPERTY_Bahrain_Saar_Compound_1.jpg</photo1>
    <photo2>906140141208_BAHRAIN_PROPERTY_Bahrain_Saar_Compound_2.jpg</photo2>
    <photo3>906140141208_BAHRAIN_PROPERTY_Bahrain_Saar_Compound_3.jpg</photo3>
    <photo4>906140141208_BAHRAIN_PROPERTY_Bahrain_Saar_Compound_4.jpg</photo4>
    </photos>
    <home_address/>
    <home_address2/>
    <home_address3/>
    <furnished/>
    <rooms>5</rooms>
    <bathrooms>5</bathrooms>
    <tv>1</tv>
    <communal_pool>1</communal_pool>
    <private_pool>0</private_pool>
    <dishwasher>1</dishwasher>
    <washing_machine>1</washing_machine>
    <microwave>1</microwave>
    <bath>1</bath>
    <shower>1</shower>
    <airco>1</airco>
    <sauna>0</sauna>
    <gym>1</gym>
    <surface_area>0</surface_area>
    <plot_size>0</plot_size>
    <sale_price>9999999999999</sale_price>
    <rent_price>2500</rent_price>
    <build_year>0</build_year>
    <rentsale>rent</rentsale>
    <description>TRULY EXECUTIVE,&#13;
    5 BEDROOM DOUBE STOREY COMPOUND VILLA.&#13;
    &#13;
    The ground floor holds the living dining, family lounge, guest bedroom.&#13;
    The first floor holds 4 nos. of bedooms with ensuite bathrooms & built in wardrobes, plus lounge/TV room.&#13;
    Also 2 servant rooms, a massive private pool, lush maintained garden, plus common pool, gym, party hall, tennis court, security-24 hours. &#13;
    Monthly rent: - BD. 2500/- &#13;
    </description>
    <agency_reference>1019</agency_reference>
    <agent_name>*****</agent_name>
    <agent_logo>test-logo.gif</agent_logo>
    <lat>26.20343</lat>
    <lon>50.48741</lon>
    </property>

    D
    Dan_Bracuk
    Inspiring
    August 22, 2011

    Regarding:

    One website is on a shared hosting where I can't install software, the other website is running on a dedicated server to which I have full access to install what I want.

    Can you move the site from you shared hosting location to your dedicated server?

    D
    Dan_Bracuk
    Inspiring
    August 22, 2011

    Instead of transferring the data, why not simply query it from the 2nd site?

    S
    s_filipowiczAuthor
    Known Participant
    August 22, 2011

    Thanks!

    I would rather not query a database over the internet.

    A
    Adam Cameron.
    Inspiring
    August 22, 2011

    I take your point, but I assume (?) JDBC connections can run over SSL, like HTTP can (ie: HTTPS)?  But security aside, it's not the fastest way of going about things ;-)

    --
    Adam

    Owainnorth
    Owainnorth
    Inspiring
    August 22, 2011

    Although I've not specifically done them myself, SQL Server (or whatever database platform you use) has some pretty powerful XML functions, I'm sure there'll be some way of extracting and loading XML natively rather than using CF, which does tend to handle large XML documents quite badly.

    Are you running 32- or 64-bit CF? This is the kind of thing that would benefit from more RAM I suspect, but it won't make a world of difference. I'd be looking into creatiing and importing the XML completely outside of CF, maybe using .NET or Java natively if there are no database tools available.

    As a quick fix for the actual transfer process - I'd have a scheduled task run every hour or so to create the master document, the latest version of which is served up by your webservice. That way when you come to do your update yes, the data might be an hour out of data but you'll get the data immediately.

    O.

    EDIT: Realised you mentioned running it every few days, clearly don't run an update every hour in that case, pick something more appropriate

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices