Android 5.0.2 Upgrade Invalidates RTMPS Streams

[Dr__Danny]

Hello,

I was the first yesterday in an unfortunately growing number of people who got an Android update which invalidated our Adobe Air app on Android only, not on iOS or the desktop as far as we can tell.  I am not exactly sure what updated except I know there was a Google Play update which updated six apps and I believe my Android version which is now 5.0.2.  Long story short, the app stopped working due to SharedObjects not working which are created using RTMPS, we get a generic "NetConnection.Connect.Failed" error code.  This is not new code or new certificates - this has been working for a couple of months.  I confirmed my certificates and all certificates in the chain did not expire.  I reinstalled my app, tried different versions of the app, tried using different servers with different hosts and certificates, rebooted the device, reinstalled Adobe Air, upgraded my Air and Flex SDKs, upgraded Oracle Java (JDK) on the server to 1.7u80 and also tried 1.8u45 - nothing seems to fix this issue and we are concerned that the Android upgrade will outpace our ability to fix it.  Since iOS and desktop apps work with RTMPS, we think the certificates are fine, but not sure.  Also, HTTPS to the same server works fine from the app, just RTMPS is not working.

Today we took a perfectly working Android device (both devices mentioned are Samsung Galaxy tablets) on Android version 4.x.y, tested it (it worked), upgraded Android to 5.0.2, tested it, and it stopped working.  So we know we are onto something with the new version breaking RTMPS.

Anyone have any ideas what to do?  Downgrading to RTMP is not an option.

Thanks,

Danny

[piyushb89979246]

Hi,

Could you please share your sample project so that we can debug this issue in a better way?

-Piyush

[Dr__Danny]

Hello Piyush,

We have spent a lot of time debugging this and finally think we figured out what is wrong, but it appears to be beyond our control to fix.  There are three areas of concern: Android Lollipop, Adobe Air, and Java. Any help is of course appreciated.

Android Lollipop - Android 5.0 Behavior Changes | Android Developers

See the link above, section, "TLS/SSL Default Configuration Changes".  MD5, 3DES, export, and static key ECDH cipher suites are now disabled.  Forward Secrecy cipher suites (ECDHE and DHE) are preferred.

Adobe Air - Adobe Flash Platform * Basics of networking and communication

Adobe Air supports SSLv3 and TLSv1.  There is not enough supporting documentation to know which ciphers Adobe is using.

Java - https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html‌

Java has explicit ciphers it supports and there are footnotes denoting the TLS compatibility for particular ciphers.  The cipher we suspect Adobe Air is using is only compatible with TLSv1.2 in Java.  Adobe Air is not stepped up to TLSv1.2, it is on TLSv1.  We are not completely sure, but we think according to the Java security log, that Adobe is trying to use, "Negotiating:  [Session-1, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]". According to the Java link above, that cipher is not compatible with TLSv1.

How do we force Adobe Air to use a different cipher, even if we are limited to TLSv1 that are in the Java compatibility list and the Android Lollipop list?

Also, getting our project is slightly complicated, we require login credentials and an account.  I can make that for you since you work for Adobe (this is not something we normally support), but you would have to contact me offline so I can create the credentials and share the project information.

Thanks,

Danny

[piyushb89979246]

Hi,

I have created a sample Project using RTMPS protocol and it is working fine on android Samsung s4(GT-I9500) 5.0.1 device.

Please check whether <SSLCypherSuite> tag in adapter.xml file is changed or not. The value that i have used is

<SSLCipherSuite>ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH</SSLCipherSuite>

Could you please check if you are facing the same issue on any other device having OS apart from 5.0.2?

Regards

Adobe AIR Team

[Dr__Danny]

Hi Piyush,

We really appreciate you trying to help us out, particularly with writing a test application.  Unfortunately the suggestion did not work because the cipher that is chosen in the handshaking process is already within those parameters.  We spent a couple of days debugging and here is what we came up with:

• We meticulously went through EVERY possible cipher, shutting down and recoding the server each time, connecting with our Adobe Air app, and monitoring the server logging.  These are the only ciphers that could work based on the logging: TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA.  They don't however work because of the combination of Adobe Air using TLS1 and the new restrictions in Android Lollipop.
• We think we understand why Adobe Air is sending us the NetConnection Failed callback - we monitored the network traffic using wireshark and observed the phenomenon documented here Issue 88313 -  android -  TLSv1.0 connection error: SSLHandshakeException errors on Android 5.0 -  A....  Since Android is not properly supporting the ciphers for TLSv1, it doesn't decode the handshake and drops the connection.  Adobe Air should be receiving an SSL exception from Android, but it only bubbles up to our application as a NetConnection failure with no additional text to help us understand what went wrong or to confirm this is the problem.  If there is a way to turn on extra debugging, please let us know.
• Android 4.x.y applications, as well as iOS and desktop do not experience this problem and work perfectly.  We wondered why iOS works - we determined that Adobe Air is actually using TLSv1.2 even though the current documentation (Adobe Flash Platform * Basics of networking and communication) says it only supports SSLv3 and TLSv1.
• We also tested this three times with three different devices - we confirm they work with Android 4.4.3.  When we upgrade to 5.0.2, at that point the devices experience this problem.  We repeated this problem on the Samsung Galaxy Tab 4 and HTC One (M8).

Since Adobe Air is supporting TLSv1.2 on iOS, is there a way to upgrade AIR to support TLSv1.2 on Android?  Even though Android claims to still support TLSv1, it appears from this error that they are putting more attention into the more modern approaches of TLSv1.1 and TLS1.2 (as they allude to here Android 5.0 Behavior Changes | Android Developers).  Is there a scheduled time when Adobe will step up to TLSv1.2 (it's been 7 years since TLSv1.2 has been released, and TLSv1 is from 1999)?

If you have any other ideas, we will certainly give them a try.

Thanks again,

Danny

[Dr__Danny]

Piyush,

I am writing to provide some additional information about us debugging this issue, but we have still not solved it.

• We setup a new separate server on a local network to ensure nothing between our app and server was getting in the way of this working (firewalls, NAT, etc.)  That did not change anything, the problem is still reproducible.
• We can reproduce the problem with a simple NetConnection test driver running on Android 5.0.2 devices.  Other than putting "rtmps" in the connect call, there is no reference to rtmps anywhere and as expected RTMPS does not work.  When we change the protocol to RTMP, it works as expected.  Side note: we also tried "http" because the documentation says that is a valid value, but we never receive an HTTP request on the server.
• We removed the server from the equation completely and simply ran tcpdump as a pseudo-RTMPS server.  This reproduces the issue, but Wireshark does not show the extra packet retransmission - possibly because we are not running a real server.  But the client side still receives NetStatus failure when running RTMPS.  When we run RTMP this way, it works.
• The code now listens to all possible event types, there are four documented: NetStatus, SecurityError, IOError, and ASynchError.  Unfortuantely, only the NetStatus failure error is encountered which does not provide any text as to what is going wrong.  We were hoping that we would see more errors by subscribing to them.  If there is a way to log more error information, we would like to know about it.
• If we do something to mess up the certificate, we do get a NetStatus certificate error.  That leads us to believe that our certificate is imported correctly.  If there are any tricks to importing tickets as seen by Android that you know of, please share.  We are aware that Apple iOS devices require the full certificate chain.
• We downloaded and installed a third-party app on our Android device called Broadcaster to test RTMP and RTMPS without using code that we wrote.  Broadcaster works fine when connecting with RTMP, does not work when connecting with RTMPS (we just get a black screen, no error).

If you or anyone reading this thread has any ideas, please let us know.  Also, if there is a public RTMPS server out there, we can point our app or Broadcaster to that IP address to see if it works from an Android 5.0.2 device.

Hopefully this information helps, we will keep trying things.

Thanks,

Danny

[piyushb89979246]

Hi Dr.Danny,

Sorry for late reply, I have debug this issue on android 5.0.2 also and it is working fine at my end. Here are the steps which I follow:

a) Set up Adobe Media Server  from Media server for streaming video | Adobe Media Server family and make it RTMPS enable.

b) Here is link of my sample project Dropbox - RTMPTEST.zip

It is working fine on all devices. Please let me know what are the changes required in this project.

Or you can share your project and credentials to me bagri@adobe.com .

Thanks

Piyush

[Dr__Danny]

Piyush,

Thanks again for helping out.  We took a long time to respond because we really wanted to get it working, but cannot, and we wanted to try a bunch of different servers to see if it is our server configuration or not.  We tried a lot of different things, I will mention what I think is useful for this thread:

• We installed AMS with default settings except we did not start it as a service, we start it manually.  We setup RTMPS according to the instructions here Adobe Media Server Help | Configuring security features.
  • The documentation references rtmps://amsexamples.adobe.com/vod/instance1, but that address does not resolve and appears to be a dead link.
• Your RTMPTEST example code is pretty much the exact same code we have except pointing to a different IP address both in our test driver and our real app.  EDIT: we noticed the example app does not set proxyType to best.  When proxyType is best, RTMPS is used.  When proxyType is not set, thus set to 'none', HTTPS-based RTMP is used (RTMPT-style).
• We setup AMS v5 x64 LS1 All, Wowza 4.2.0, Red5 1.0.4 and Red5 1.0.5 and observed the following behavior with RTMPS:
  • The avplay command works with RTMPS (with all servers).
  • The Broadcaster app works with RTMPS with AMS and Wowza.  Note: we were able to determine that even though we specified RTMPS when paired with Red5, Broadcaster was actually using RTMPT over HTTPS because of the POST command issued after the SSL handshake (shown in log).  Our Red5 server is not setup to use RTMPT which is why Red5 does not work (we assume further confirming our theory about using RTMPT).
  • One time tonight only (fortunately we catpured it), we got a log in the AMS edge log when trying to connect our test driver app with RTMPS.  It says: 2015-06-25        00:19:06        3690        (e)2711336        SSL accept failed : (err=1, ip=192.168.1.110) : error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request.  We don't know why the log is not repeatable.  Our Red5 log with full debugging says the handshake failed with no other information we can determine is useful.  We are not seeing logs specific to the RTMPS failure in Wowza.
  • Everything said before still appears to be true - Android 4.x devices work fine, Android 5.0.2 does not.
  • There was no noticeable difference between Red5 1.0.4 and 1.0.5 other than we used different versions of Java and neither worked.
• Depending on your answer, we can setup a test account if we need to.  Hopefully the SSL23 error helps figure this out.
• We are not sure if this information helps you, but we are running our server on CentOS 6.6 using OpenSSL 1.0.1e-30.  We are wondering what differences exist between our servers.

Any advice?

Thanks,

Danny

[Dr__Danny]

Hello everyone,

Providing an update.  Piyush and I coordinated offline to do a test where I setup a server and used a test driver to test.  I will send Piyush instructions how to test with our server in a bit, but here are the results.  The test driver uses an RTMPS stream but does not set proxyType to 'best' on the network connection, it uses the default proxyType 'none'.  In this mode, HTTPS-based RTMP is used instead of RTMPS.  The good news is that HTTPS-based RTMP is a secure protocol.  The bad news is it did not perform well for our app.  There are two problems: (1) the performance of the video FPS is slow and (2) the technique was not working cross platform for us.  For example, if a Mac laptop would connect to an Android camera, the screen was black on the Mac of the Android's picture.  If the Android connected to the Mac's camera, the video was visible for one second, but then the NetConnection would close after about one second.  Because of these issues, we still feel it is necessary to get RTMPS working.  When we set ProxyType to best in the test driver, newer Android 5.x Lollipop devices do not work (which is the original problem).

In the middle of all of this, we hired an expert consultant to help us out.  His conclusion is that it appears that the Android Lollipop client is closing the connection.  He could not tell if the problem is occurring from the Android OS or Adobe Air, but he knows the handshake is failing from that side of the connection based on the hex codes sent from the client during the handshake process.  I showed him the link I posted above (Android 5.0 Behavior Changes | Android Developers) where Google claims that Android Lollipop will break "a small number" of older TLSv1 ciphers.  We also ran 56 different cipher tests (only enabling one cipher at a time), hoping that one would work, but unfortunately none of the ciphers worked.  The consultant wanted me to ask two questions in this thread:

• Does the Flash Player wrap its data in SSLv2?
• Does Flash Player send or expect "StartTLS"?

Our CEO also wanted me to mention that we are developing a secure HIPPA compliant app, and in light of various Information Assurance issues in the industry, security is a top priority for us.  We appreciate your help to date. 

Thanks,

Danny

[Guest]

Hi Danny,

Just wanted to say I'm watching this thread with a lot of anticipation, as I've seen a similar issue with Android 5.0.2 (paired w/ Red5 1.0.5 in my case.) Web + Flash Player and Android 4.4.2 (Samsung Galaxy Tab 4) + AIR both work fine, but Android 5.0.2 (Moto G gen 2) sees the connection immediately close. If it helps, the negotiated cipher in both cases is TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA. I tried disabling ECDHE and DHE on the server but it has done no good, the problem seems independent of the cipher.

It's such a shame because native RTMPS is so much more performant than RTMPT over SSL, and I can't get RTMPE to work either.

Incidentally, we're also working on a HIPAA-compliant app

Dave

[savichris]

I have come across this same issue, is there any update or resolution?

[dqh360]

We have the same issue also. Submitted a bug report back in April, but no-one has looked at it.

Bug#3977064 - RTMPS proxyType "best" not working in Android 5

Makes me wonder if SecureSocket works on Android 5 or 6