Hypriot blog

Lately I have been facing a challenge, consisting in the development of a real time kernel for the Raspberry Pi, with the aim of integrating it into the Hypriot Project.

See it in GitHub.

Download the debs for installing the RT kernel from RT (follow the README from Github to install).

It is quite strange that I haven’t found information for the new kernels, like the 4.1 version that I have used.

There are some problems (seeing Chubb and Song) with IRQs when running cyclictest in order to benchmark the system. It “stalls” because “starved jiffies” and hangs. For example:

chrt -f 88 cyclictest -t1 -m -n -a0 -p 88 -i 400 -n -l1000000

So if RT Priority is greater than 50 the test blocks softIRQs:

$  ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm --sort=+pcpu | grep FF
PID  TID CLS RTPRIO  NI PRI PSR %CPU STAT WCHAN          COMMAND


2 12 FF 99 - 139 0 0.0 S posix_cpu_time posixcputmr/0
14 14 FF 99 - 139 0 0.0 S smpboot_thread migration/0
15 15 FF 99 - 139 1 0.0 S smpboot_thread migration/1
18 18 FF 99 - 139 1 0.0 S posix_cpu_time posixcputmr/1
21 21 FF 99 - 139 2 0.0 S smpboot_thread migration/2
24 24 FF 99 - 139 2 0.0 S posix_cpu_time posixcputmr/2
27 27 FF 99 - 139 3 0.0 S smpboot_thread migration/3
30 30 FF 99 - 139 3 0.0 S posix_cpu_time posixcputmr/3
38 38 FF 50 - 90 0 0.0 S irq_thread irq/65-3f00b880
55 55 FF 50 - 90 1 0.0 S irq_thread irq/49-3f200000
56 56 FF 50 - 90 1 0.0 S irq_thread irq/50-3f200000
57 57 FF 50 - 90 1 0.0 S irq_thread irq/16-bcm2708_
62 62 FF 50 - 90 1 0.0 S irq_thread irq/66-VCHIQ do
70 70 FF 50 - 90 1 0.0 S irq_thread irq/77-DMA IRQ
71 71 FF 50 - 90 0 0.0 S irq_thread irq/20-DMA IRQ
72 72 FF 50 - 90 3 0.0 S irq_thread irq/84-mmc0
591 591 FF 50 - 90 1 0.0 S irq_thread irq/83-uart-pl0
1322 1322 FF 99 - 139 2 0.0 R+ - ps
1323 1323 FF 99 - 139 1 0.0 S+ pipe_wait grep
1033 1033 FF 99 - 139 2 0.1 S wait bash
67 67 FF 50 - 90 3 0.3 S irq_thread irq/32-dwc_otg_
66 66 FF 50 - 90 0 0.4 S irq_thread irq/32-dwc_otg
68 68 FF 50 - 90 1 0.7 S irq_thread irq/32-dwc_otg_
3 3 FF 1 - 41 0 1.5 S smpboot_thread ksoftirqd/0
17 17 FF 1 - 41 1 1.5 S smpboot_thread ksoftirqd/1
23 23 FF 1 - 41 2 1.5 R - ksoftirqd/2
29 29 FF 1 - 41 3 1.5 S smpboot_thread ksoftirqd/3
1316 1316 FF 88 - 128 1 7.0 SLl+ hrtimer_nanosl cyclictest


For testing and stopping anything, it is interesting to start a real time bash:

sudo chrt -f 99 bash

When it does not have intensive I/O, it runs well:

chrt -f 88 cyclictest -t1 -m -n -a0 -p 88 -i 400 -n -h 400 -l1000000 > h160120.txt

# Total: 001000000
# Min Latencies: 00014
# Avg Latencies: 00016
# Max Latencies: 00097
# Histogram Overflows: 00000
# Histogram Overflow at cycle number:
# Thread 0:

Thank you to:

http://honglus.blogspot.com.es/2011/03/understanding-cpu-scheduling-priority.html

http://kb.digium.com/articles/Configuration/How-to-perform-a-system-latency-test

https://www.osadl.org/Realtime-Preempt-Kernel.kernel-rt.0.html#externaltestingtool

http://kb.digium.com/articles/Configuration/How-to-perform-a-system-latency-test

EDIT 2016-02-12 Friday:

Well, it seems that the hangs come from this: https://wiki.linuxfoundation.org/realtime/documentation/known_limitations

Also see https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=159170