- This topic has 6 replies, 2 voices, and was last updated February 3 by
.
-
Topic
-
Hi, Youssef.
Hope you are doing well. I have some time to compare performance on v1.1.0 vs 1.2.0 on GPU and strange staff happened.
Before you published 1.2.0 with example to work on GPU I managed to figure out how to do it with v 1.1.0 and I got expected results – 15-30 ms for GPU step_time and 80-90ms for CPU.
Both GPU/CPU tended to converge accordingly (increase accuracy and decrease loss)
After upgrading to v1.2.0 more or less same cycle time was demonstrated, but in attached example with GPU activation loss function convergence did not occurs.
due to the fact that it is just GPU/CPU functionality, I am not sending you VI – it is the same as per example, (just GPU/CPU switch change to visible).
I also have video taped screen – let me know if you’d like to get it – too big to attach to mail.
Best regards and good luck
Vlady Kaplan
-
Replies
-
Dear Vlady,
A new version arrives today (1.2.0.8). We are waiting for the end of its creation is which is in progress at the moment (As HAIBAL has more than 4000 VI it takes more than 4 hours to each build to be done …).
Apologies for the lack of video guide to switch between CPU / GPU (did not have time to do it and it will be done today for future users since you now know how to do)
Once the new version is released, can you test with it? If the problem persists, we will start looking for this bug. (there will be a lot of changes on this version that should have been called 1.2.1 but it is not important)
I am preparing a release note for the new version (1.2.0.8).
Thank you for your feedback !
https://jumbomail.me/j/oNu1e5cdZEqdoJn
I just would like to notify that for first version GPU was working properly, no issues were detected with convergence.
Thanks
Vlady
We found problem. It is a problem of architecture (if you look at your state QMH FIFO you’ll see it). When we change GPU /CPU or CPU /GPU the test thread continu to send FIFO message to Train thread that create this problem.
We made modification to avoid this problem. In next release MNIST will be updated.
🙂
Theoretically this problem did not exist because we were not supposed to play with the architecture. But it’s important that some people play with HAIBAL, we could have run into a bug.
We had internally a reflexion if you are agree to continu to find bugs or problems we will give you more licence time 🙂
It is important for us to test our system to gain credibility.
Thanks Vlady
- You must be logged in to reply to this topic.