BC issues after migration

Report · Oct 24, 2013

Hey all,

I know evidence needs to be put up in a lot of these cases and we at Pretty will try get them but we been super busy this week and trying to work around these issues at the moment. Speaking to others, some BC staff and other partners other people are having these issues:

Admin:

1. Slow admin. Amazon supposed to make things nice and more speedy but overal the admin for AU sites at least is noticebly slower.

2. BC admin element loading lag. This is quite bad in Firefox, better in Chrome but you can click on editing of a custom field in a web app and it will just lag for a long time or just have the load spinner locked. Same thing for managing categories or if you go into custom fields etc.

3. Some Australia sites web pages have no archive and roleback history, some do but some have nothing for 90days even if a client has edited them.

Partner portal:

1. Seems some cases of expired sites not showing up, This may be down to their expiry time and you guys cleaning them out during migration?

2. Partner portal is very slow

File loading slow:

A lot of the BC files from the CDN side of things I was hoping would speed up after the migration and things like the AU proxy was cusing these to be slow recently but after migration this has not changed or got worse.

You can look at console and see some of the BC scripts etc taking 1.2 to 2 seconds to load and all files seem to have millisecond access delays fetching from the server before they are loaded.

Caching issues:

1. Back with a vengence after migration, uploading CSS files or JS files and they not updating and doing so a few times, clearing browser cache and just having to wait for BC to accpet the new file.

2. I actually have a couple of examples (shown a few people) of images that have been replaced, deleted and replaced and if you sit there and reload the file many times you can get the image from the previous day or 2 days ago version show up.

File deletion / Corruption issues:

Tim was mentioning if I had issues with apps and files and I never twigged on what those could be and I also was writing files and wondering why they were not showing up in apps. Then Matt here at Pretty was getting courrupted files, messed up CSS files through DW SFTP uploads as well.

I was also getting upload errors in Coda and having to keep putting files till it uploaded.

I have heard cases with other partners and BC site as well ( the podcast mp3 vanished) where files have just been removed. They likely on the system still but the reference to it goes and the file can not be found.

We will try and post screenshots and videos when we can of these issues and I hope others can also post here their findings to help you guys sort these issues.

With the migration out the way and while I know you guys are busy I would really like you guys to look to see if you can turn off the old admin yet?

You guys in the past have said some of the new UI slowness is as a result of this but for me it looks like you need to still do more work as some of the new UI seems to be script/CSS skinning of the old UI and that still needs to be there to allow some of the admin to function and why it seems slow.

Report · Oct 24, 2013

I will put this here as well for everyone to comment on in terms of actually logging into the BC admin.

Has anyone had a failed login (this is before the migration as well) where you are 100% certain you have entered the right password, you have done so a couple of times and then then only the third or fourth time it lets you in and its the same password?

Speaking to a few people I am not the only one that has experienced this.

Report · Oct 24, 2013

We actually faced this issue in some of our computers, but we found a fix to it - not sure if this would solve everyone's issue. We are on Macs - what we had to do is delete all the saved login details within Keychain - and all Macs were fine to login again.

Report · Oct 24, 2013

Hey Liam - yeah we have the issue of expired websites not appearing - have raised to Florin

Report · Oct 24, 2013

We haven't got that far in details with these issues but yes I can confirm some of them.

Caching - OMG - it's doing our heading - we still get errors even after we fix them. As an example, we create a function, when the browser detects the error or simply doesn't render it, we fix it, and we also change the variables in the function, but the browser detects the error pointing at the old variables that we replaced. That's just mad.

But in terms of the Admin itself, I must say that before migration, the Admin was performing bad but better than Now - if that makes sense. Lags, Slowliness, freeze ... are just common at this stage. we added an extra use while we are in Admin (Cmd+R) refresh all the time. Web Apps manipulation is just a nightmare at the moment.

I just hope that those issues can be fixed soon before we start getting serious complaints from clients.

Report · Oct 24, 2013

Hey guys, could you also look at your XML files and how they rendered in the Amazon environment as well:

http://www.prettypollution.com.au/RSSRetrieve.aspx?ID=5304&Type=RSS20

Also seeing simiar errors with product feeds. The two I have encounted are live sites i had to fix up and just edited out some of the inline html but some things like this:

Both chrome and ff validate xml feeds now and these showing issues.

These were working fine last week.

Report · Oct 25, 2013

The http://www.prettypollution.com.au/business-catalyst-blog/surfrider-the-ocean-and-its-hereos blog post contains special control characters not valid in XML even when encoded, 

Find the should instigate in text in the blog post and execute escape('should instigate in') in the browser's JavaScript console to see the otherwise invisible character with ASCII code 3 escaped as %03.

Report · Oct 25, 2013

Just written in the BC blogs and product feed issues clients doing content in the feeds.
As I mentioned in all these cases these were working last week and now are not. So something that was escaping isnt now or something along those lines.

Report · Oct 25, 2013

Could it be possible that you didn't notice the issue until after the migration?

The issue with invalid XML in the feeds caused by certain special control characters is old, we made no changes in this area, the migration didn't affect this.

There are 2 blog post in the Pretty Polution site that contain the ETX character (ASCII code 03) that causes the issue,

"Pretty's New Handshake" and "Surfrider. The ocean and its hereos."

Could you provide in a private email the products body content from the other sites where you had issues with the rss feed? Just to identify troubling characters.

Thanks,

Dan Popa

Report · Oct 25, 2013

Hey guys,

Let's take these one by one. As you've mentioned Liam, we might need more details on some of them, see below. Please use the PAB address (pab@bcsupport.zendesk.com) to file separate tickets for each of the items where we might need additional details.

Admin:

1. Slow admin. Amazon supposed to make things nice and more speedy but overal the admin for AU sites at least is noticebly slower.

2. BC admin element loading lag. This is quite bad in Firefox, better in Chrome but you can click on editing of a custom field in a web app and it will just lag for a long time or just have the load spinner locked. Same thing for managing categories or if you go into custom fields etc.

[Florin] Engineering is looking into this - we might need more details or traces of what you see/get in Admin. We'll come back with more details

3. Some Australia sites web pages have no archive and roleback history, some do but some have nothing for 90days even if a client has edited them.

[Florin] I need more details on this one - which site, which pages

Partner portal:

1. Seems some cases of expired sites not showing up, This may be down to their expiry time and you guys cleaning them out during migration?

[Florin] I'm looking into this now.

2. Partner portal is very slow

[Florin] Engineering did a full investigation on Partner Portal speed earlier this week, and several improvements have been put in place, issues have been fixed. Are you still seeing slow performance in Partner Portal? Does this happen for other partners as well?

File loading slow:

A lot of the BC files from the CDN side of things I was hoping would speed up after the migration and things like the AU proxy was cusing these to be slow recently but after migration this has not changed or got worse.

You can look at console and see some of the BC scripts etc taking 1.2 to 2 seconds to load and all files seem to have millisecond access delays fetching from the server before they are loaded.

[Florin] This will be looked at as part of the Admin performance

Caching issues:

1. Back with a vengence after migration, uploading CSS files or JS files and they not updating and doing so a few times, clearing browser cache and just having to wait for BC to accpet the new file.

2. I actually have a couple of examples (shown a few people) of images that have been replaced, deleted and replaced and if you sit there and reload the file many times you can get the image from the previous day or 2 days ago version show up.

[Florin] Again, this is one of those cases in which we need more details. Please email the PAB address with some steps we can take to replicate this.

File deletion / Corruption issues:

Tim was mentioning if I had issues with apps and files and I never twigged on what those could be and I also was writing files and wondering why they were not showing up in apps. Then Matt here at Pretty was getting courrupted files, messed up CSS files through DW SFTP uploads as well.

I was also getting upload errors in Coda and having to keep putting files till it uploaded.

I have heard cases with other partners and BC site as well ( the podcast mp3 vanished) where files have just been removed. They likely on the system still but the reference to it goes and the file can not be found.

[Florin] The file corruption issue that you and Matt were getting have been ruled out to be caused by a temporary connectivity issue between our webservers and the disks. The files weren't written properly, but the links to them were created in the database. That's why you couldn't access them later on.

You shouldn't encounter this problem anymore for new files created on that site.

The issue with the podcast mp3 file was another issue, specific to the businesscatalyst.com site - and it's been fixed.

[Florin] if you or Matt, or any other PAB partner get any issues uploading/retrieving files, our recommandation is to file a PAB ticket with the site and problem details and we will investigate further.

Thanks and regards,
Florin

Report · Oct 25, 2013

I have on some of those florin and things get shut down etc. I was saked to take them into a thread on PAB now im being asked to split things up again. It gets a bit confusing on what supposed to do. Some of you say ticket them then that gets lost in the system, get told some things could be done in emails and no replies or PAB email which gets moved etc.

Dan, no, Brett does a lot with the SEO and feeds etc and was not working.

Google product feed wise I been working on a few things in regard to these so rendered and viewed several feeds over the last couple of weeks and they erroring.

I did not include the module_subscriptions JSON having an error for example as I think thats just a mistake in coding as its invalid JSON.

Ill go email again guys. Ill also provide file console net info as well. CSS files with only one line of code taking nearly a second to fully render for me is a bit much. There seems to be a lot of blocking then waiting on files.

Report · Oct 27, 2013

Florin, The issues we are highlighting occur randomly between multiple sites. It is hard for us to specify which error occur on which site as they vary depending on time, site and ocurrance itself. I have also noticed some old fixes that have been broken again after migration. One of them is ICE and style rendering. If you remember I have raised this same issue when ICE was still in Beta and you have fixed it in the release after that. Well, now it's back.

The issue is that when you have a content holder in the <head> Tag (which contains js, css files and what have you...) the page doesn't render properly - meaning no styling gets applied.

Well... after a quick look... actually no content holders gets rendered whatsoever.

......

Ok after another trial, Named Content Holders don't render

Report · Oct 28, 2013

Hey Tarik.

Regarding the named content holders that don't render: can you please send Florin the info of the Site, Page & Content Holder that doesn't render (by name)?

I have tried on several sites and the named content holders work (ex: {module_contentholder,name="contentholder1"}). Can you send us some info so that we can duplicate?

Thanks,

-Lucian

Report · Oct 28, 2013

To add to this, we are finding after migration Analytics that is in a content holder in the head is not being picked up by google any more.

Further to this I want to get some time for you to re-test these issues:

1. Javascript looking for an element in the layout that is just javascript called inline in the head cant find targeted elements in layouts.

2. Without wrapping everything in a head and body and full html for every layout if you run social media validation tools like Facebooks own validation and you have a like button to a detail page thats in a list view these do not work correctly. There seems to be a lag to rendering.

3. Content holders as head items render to slow to cause issues as mentioned.

In all these cases there seems to be what may be millisecond delays to layout and content holder rendering that is slower then these scripts from social media to google anayltics to basic root javascript from seeing elements loaded into the dom. It is like BC is rendering them milliseconds to slow.

Report · Oct 28, 2013

Although the physical migration for Amazon has completed, the project is still in optimization phase and our engineering team is still working on ironing out the new infrastructure to perfectly fit BC.

As a result, some issues may still appear while we are working on the new infrastructure layout, but we will do our best to minimize customer and partner impact.

With regards to your specific items:

Some sites don’t have archive and rollback history - this issue has been fixed. If you continue experiencing problems please contact our support team
Expired sites do not show up – This issue has been fixed and they should all show up, please report to support in case they don’t
Caching issues – we are unaware of any caching issues, everything should be invalidated instantly. If you can submit a case with details about site ID and replication please help us identify this one.
File deletion / corruption issues – again, we did have some reports but nothing can be reproduced.

We do apologize for the slowness you are experiencing, and can assure you our team is looking into these issues and finding solutions for them as we move further. We have new logging enabled in the AdminConsole that will give us a browser based loading reports based on the geography.

We expect the optimization project to continue within the next month and will post a complete update when this is finished.

Your feedback is deeply appreciated, as it will show us the items where we still need to work to bring things to the best possible solution. Please continue to provide this through support cases by emailing pab@bcsupport.zendesk.com and do not forget to mention the Site ID and network tab (in your browser developer console to allow us to see the times in which assists are served) so that it is easier for our engineering team to debug the issues.

Thank you for all your help and we appreciate your patience with this ongoing project.

Magda

Report · Oct 28, 2013

Lucian, the content holders DO render in the site front-end but DO NOT render using ICE. I think this issue occured at launch of the new ICE and has been fixed. But it's back again.

Report · Oct 28, 2013

Lucian, I just sent an email to Florin. Can you guys please review that issue as it is kind of urgent for some of our current projects.

Thanks.

Report · Oct 28, 2013

Caching issue at work:

This file has this and this was uploaded succesfully:

This is the online version after a hard refresh that should clear and update the file.

Not updated, After a couple of clearing cache using webdeveloper toolbar and command R's on the mac it eventually updates.

In terms of file uploading I am still seeing varius errors as well:

Report · Oct 29, 2013

Hi Liam,

I've escalated this one to our engineering team as well. Thanks for posting all the details - I'll update you as soon as I have any news from the engineering team.

Thanks,
Florin

Report · Oct 31, 2013

Hi everyone,

Our engineering team is working on the slowness issue, and it would really help if you can provide some detailed information on this. Can you please follow the steps below and send the resulting data to our engineering team?

To help us investigate slowness in Admin Console send us an export of the NET panel from Firebug.

Install NetExport Firebug extension in Firefox from https://getfirebug.com/releases/netexport/netExport-0.9b3.xpi
Login into you Admin Console using Firefox
Navigate to the Admin Console page where you experience slowness.
Delete the browser cache to force reload all resources; it's enough to just delete the disk cache.
Open Firebug, make sure the NET panel is enabled.
Reload the Admin Page
Export the NET panel activity using the Export button provided by the NetExport Firebug extension, the "Save as JSONP..." option.
Email the .har file to Dan Popa (dpopa@adobe.com) for inspection.

Thanks,

Magda

Report · Oct 31, 2013

Will do Magda.

Report · Nov 01, 2013

I will have some reports for you gues Tuesday, I got to do some work on the weekend so I wanted to give you some different days so you have some varied data.

Report · Nov 03, 2013

Just for your feedback we are increasingly getting complaints from Clients on the slowness of BC over last week and already this morning in a new week/over the weekend.

Report · Nov 14, 2013

Hey guys,

How is this going? I have sent a few file reports, got some more ready to go as well

But things are getting slower, Web apps specifically - Anything related to them are just running very slow.

You can see just from here:

https://www.dropbox.com/s/ayqd7s2ie4j6f6u/Screenshot%202013-11-15%2009.51.47.png

9 Seconds to save a web app item with only 4 custom fields, pretty basic on a new site.

Beacon-2.newrelic.com I have seen last few days specifically take long processing times and anything that pops up from api-asyd.worldsecuresystems.com is really slow as well.

This morning also seeing very very slow FTP, folders taking several seconds to populate and FTP errors:

https://www.dropbox.com/s/0telz4vq0xi3pgz/Screenshot%202013-11-15%2009.59.22.png

Really concerned about BC running on Amazon, do you think you guys be able to address these increasing issues?

Report · Nov 15, 2013

Hi Liam,

The slowness you've noticed yesterday was due to an incident we had on AU datacentre, which is now closed. So, while the initial problems are still beeing looked on, yesterday degradation was not related to the migration or admin itself.

Best regards,

Cristinel

Adobe Community

BC issues after migration