iOS keychain cannot be accessed in unit tests

[odrobnik]

A unit test accessing the iOS keychain fails when run via xctool but works when run via xcodebuild.

This issue has first been described here

Here are my own tests, look at the bottom of the build log:

• with xctool, failure "No keychain is available. You may need to restart your computer"
• with xcodebuild: the tests pass

[LegNeato]

Yeah, this is likely because we don't run unit tests in the simulator context. Apple has been moving this direction for a bit now and we plan to go that way too.

[odrobnik]

Pray tell how to do that

[LegNeato]

It will require some xctool changes unfortunately. This was a change Apple did in the underlying system in Xcode 5. We have it on the todo to run unit tests in the simulator context, which will solve issues like this.

As a workaround, you can either swizzle or turn the test into an integration test (which runs in the simulator).

[odrobnik]

Please explain how to "turn the test into an integration test".

[grp]

I think @LegNeato means an application test, which means you'd set the TEST_HOST and BUNDLE_LOADER build settings to an app to run the test inside of. If it's just for this, you can make an empty app for that.

[LegNeato]

Yeah, sorry I used the wrong terminology!

[ExtremeMan]

Fixed since we have switched to use simctl to run logic tests (6b59d20). I've checked against the https://github.com/danielalves/NitroKeychain project. All tests are passed.

[lordkev]

I'm still getting "No keychain is available. You may need to restart your computer" with 0.2.7. Tests run fine using xcodebuild.

@ExtremeMan Is there something else I need to do to get it working with the switch to simctl?

[ExtremeMan]

Hm. Are your tests application tests, @lordkev?

[lordkev]

@ExtremeMan: No, unit tests.

[ExtremeMan]

@lordkev, can you try moving the test to application tests? (xcodebuild is launching tests in Simulator.app context, but xctool for performance is not doing that for logic tests).

[lordkev]

@ExtremeMan I'll give it a shot, but I'm kind of confused about your comment from Aug 13 about simctl fixing the issue and the tests on the NitroKeychain passing. Was that only when changing them to run as application tests?

[ExtremeMan]

Hm, my comment is about logic tests, true. I will need a sample project which I can use to test. Can you help me with that?

[lordkev]

Just realized - in our case it's tests for a framework, so there isn't really any application to host it. NitroKeychain is still a good test project. I just verified that with xctool 0.2.7 three of 16 tests fail, but with xcodebuild all tests pass.

[ExtremeMan]

Ok, I will try it out.

P.S. You can always create an empty test host app for your framework.

[ExtremeMan]

I looked into it. Yes, NitroKeychain tests are failing. Even if I checkout the xctool revision dated when I closed this issue. I think it is related to Yosemite security changes. Nevertheless the easy fix exists - run those test within application context. I have created an empty test host application for the NitroKeychain test bundle and it just works now.

I will probably need to keep this issue open since Xcode and xcodebuild apparently can run those tests somehow. Most likely they are forwarding keychain requests to simulator app? I am not sure and I am not sure if it is worth investigating since it could be easily fixed.

[ExtremeMan]

Ah, that is interesting. After I reset my changes to NitroKeychain project (converted back to logic tests) they are passing. If I kill simulator app - they are failing. It might be related to some services triggered by Simulator app on launch.

[ExtremeMan]

And if the simulator device is changed to the one that doesn't match the one used by xctool (for example, I open "iPhone 5" device in Simulator.app and specify "iPhone 4" when running tests via xctool) - tests fails again. So it is definitely related to Simulator.app running at the same moment. And simulated device should match.

[diogoguimaraes]

I'm facing the same problem and the workaround I'm using is described in the following steps:

• Check which device travis is using during tests, you can find it in the log right when travis begins the Xcode tests;
• Run xcrun simctl listto check the device's UDID;
• Open the simulator with the device's UDID before running xctool (before_script):
  - open /Applications/Xcode.app/Contents/Developer/Applications/Simulator.app --args -CurrentDeviceUDID <UDID>;
• Call xctool with your default configurations.

It would be great to have xctool handling this in a seamless way.

[ExtremeMan]

@diogoguimaraes, thanks for the notes! I will have a look if xctool could launch a simulator app before running logic tests. Should be doable =)

[emimh]

Any update on this issue? Still having this problem in 0.2.9 ...

[ExtremeMan]

As a workaround just convert tests to application tests or launch simulator manually.