0

Introduction:

I am running a cpp application on an embedded systems device with onboard Linux, which gathers sensor data, prepares it into a JSON formatted string payload and publishes the payload on a topic to AWS IoT Core. This routine happens every 1 minute.

For debugging purposes we are printing the payload on the serial console before sending it, along with a lot of other console log messages. We use a FTDI C232HD-DDHSP-0 console cable connected to the UART Tx/Rx pins of the embedded board and we estabish a serial connection using PuTTY to view the log messages.

My embedded system configuration is as follows

System Parameters

U-Boot 2016.03-g80540a2-dirty (Mar 15 2019 - 16:43:43 +0800)

CPU:   Freescale i.MX6ULL rev1.1 528 MHz (running at 396 MHz)
CPU:   Commercial temperature grade (0C to 105C) at 46C
Reset cause: POR
Board: MYS-MX6ULL 14x14
I2C:   ready
DRAM:  256 MiB
force_idle_bus: sda=0 scl=0 sda.gp=0x1d scl.gp=0x1c
force_idle_bus: failed to clear bus, sda=0 scl=0
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
Card doesn't support part_switch
MMC partition switch failed
*** Warning - MMC partition switch failed, using default environment

Problem:

The device is run continuously for long periods of time as part of our testing. The application running on the device promptly sends the data every minute to the cloud in the first few hours of the test run.

After a few hours no data is received on the cloud side. When we open a serial terminal through PuTTY to debug the device, the console does not respond to any keystrokes we enter. When we close and reopen the serial terminal, the log messages are printed continuously while the some of the payload is also sent to the cloud successively, flooding the terminal window till the point where the application has no more log messages to print. After this, the application is running normally. The dmesg output that I got at this state is as below

Booting Linux on physical CPU 0x0
Linux version 4.1.15+ (sacrasystems@ubuntu) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #1 SMP PREEMPT Mon May 20 10:11:32 IST 2019
CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: Freescale i.MX6 ULL 14x14 EVK Board
Reserved memory: failed to allocate memory for node 'linux,cma'
Memory policy: Data cache writealloc
On node 0 totalpages: 65536
free_area_init_node: node 0, pgdat 80c10cc0, node_mem_map 8fdf7000
  Normal zone: 512 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 65536 pages, LIFO batch:15
PERCPU: Embedded 12 pages/cpu @8fdd4000 s16908 r8192 d24052 u49152
pcpu-alloc: s16908 r8192 d24052 u49152 alloc=12*4096
pcpu-alloc: [0] 0
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
Kernel command line: console=ttymxc0,115200 fstype=ext4 root=/dev/mmcblk0p2
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 246888K/262144K available (8509K kernel code, 439K rwdata, 2948K rodata, 428K init, 450K bss, 15256K reserved, 0K cma-reserved, 0K highmem)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
    vmalloc : 0x90800000 - 0xff000000   (1768 MB)
    lowmem  : 0x80000000 - 0x90000000   ( 256 MB)
    pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
    modules : 0x7f000000 - 0x7fe00000   (  14 MB)
      .text : 0x80008000 - 0x80b386e4   (11458 kB)
      .init : 0x80b39000 - 0x80ba4000   ( 428 kB)
      .data : 0x80ba4000 - 0x80c11ea0   ( 440 kB)
       .bss : 0x80c14000 - 0x80c84b00   ( 451 kB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Preemptible hierarchical RCU implementation.
        Additional per-CPU info printed with stalls.
        RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=1.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
NR_IRQS:16 nr_irqs:16 16
mxc_clocksource_init 24000000
Switching to timer-based delay loop, resolution 41ns
sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
clocksource mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
Console: colour dummy device 80x30
Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=240000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
/cpus/cpu@0 missing clock-frequency property
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x80008280 - 0x800082d8
Brought up 1 CPUs
SMP: Total of 1 processors activated (48.00 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
device-tree: Duplicate name in lcdif@021c8000, renamed to "display#1"
VFP support v0.3: implementor 41 architecture 2 part 30 variant 7 rev 5
clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor ladder
cpuidle: using governor menu
hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 8 bytes.
imx6ul-pinctrl 20e0000.iomuxc: initialized IMX pinctrl driver
imx6ul-pinctrl 2290000.iomuxc-snvs: initialized IMX pinctrl driver
mxs-dma 1804000.dma-apbh: initialized
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
i2c i2c-1: IMX I2C adapter registered
i2c i2c-1: can't use DMA
media: Linux media interface: v0.10
Linux video capture interface: v2.00
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
MIPI CSI2 driver module loaded
Advanced Linux Sound Architecture Driver Initialized.
Bluetooth: Core ver 2.20
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP socket layer initialized
Bluetooth: SCO socket layer initialized
Switched to clocksource mxc_timer1
NET: Registered protocol family 2
TCP established hash table entries: 2048 (order: 1, 8192 bytes)
TCP bind hash table entries: 2048 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 2048 bind 2048)
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
imx rpmsg driver is registered.
Bus freq driver module loaded
futex hash table entries: 256 (order: 2, 16384 bytes)
VFS: Disk quotas dquot_6.6.0
VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered
jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
fuse init (API version 7.23)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
imx-weim 21b8000.weim: Driver registered.
backlight supply power not found, using dummy regulator
MIPI DSI driver module loaded
MIPI DSI driver module loaded
21c8000.lcdif supply lcd not found, using dummy regulator
mxsfb 21c8000.lcdif: failed to find mxc display driver
Console: switching to colour frame buffer device 60x34
mxsfb 21c8000.lcdif: initialized
imx-sdma 20ec000.sdma: no event needs to be remapped
imx-sdma 20ec000.sdma: loaded firmware 3.3
imx-sdma 20ec000.sdma: initialized
2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 18, base_baud = 5000000) is a IMX
console [ttymxc0] enabled
21e8000.serial: ttymxc1 at MMIO 0x21e8000 (irq = 226, base_baud = 5000000) is a IMX
21ec000.serial: ttymxc2 at MMIO 0x21ec000 (irq = 227, base_baud = 5000000) is a IMX
imx-rng 2284000.rngb: iMX RNG Registered.
imx sema4 driver is registered.
[drm] Initialized drm 1.1.0 20060810
[drm] Initialized vivante 1.0.0 20120216 on minor 0
brd: module loaded
loop: module loaded
CAN device driver interface
2188000.ethernet supply phy not found, using dummy regulator
pps pps0: new PPS source ptp0
fec 2188000.ethernet (unnamed net_device) (uninitialized): Invalid MAC address: 00:00:00:00:00:00
fec 2188000.ethernet (unnamed net_device) (uninitialized): Using random MAC address: 1a:e7:44:b4:fa:22
libphy: fec_enet_mii_bus: probed
fec 2188000.ethernet eth0: registered PHC device 0
PPP generic driver version 2.4.2
PPP BSD Compression module registered
PPP Deflate Compression module registered
PPP MPPE Compression module registered
NET: Registered protocol family 24
SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256) (6 bit encapsulation enabled).
CSLIP: code copyright 1989 Regents of the University of California.
SLIP linefill/keepalive option.
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-mxc: Freescale On-Chip EHCI Host driver
usbcore: registered new interface driver cdc_acm
cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
usbcore: registered new interface driver usb-storage
usbcore: registered new interface driver usb_ehset_test
2184800.usbmisc supply vbus-wakeup not found, using dummy regulator
2184200.usb supply vbus not found, using dummy regulator
ci_hdrc ci_hdrc.1: EHCI Host Controller
ci_hdrc ci_hdrc.1: new USB bus registered, assigned bus number 1
ci_hdrc ci_hdrc.1: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
mousedev: PS/2 mouse device common for all mice
input: 20cc000.snvs:snvs-powerkey as /devices/platform/soc/2000000.aips-bus/20cc000.snvs/20cc000.snvs:snvs-powerkey/input/input0
tp reset gpio=131
msg ft5x0x_i2c_rxdata i2c read error: -5
ft5x0x read ID error!
ft5x06_ts: probe of 1-0038 failed with error 161
ft5x0x_ts_init successful
input: iMX6UL TouchScreen Controller as /devices/platform/soc/2000000.aips-bus/2040000.tsc/input/input1
snvs_rtc 20cc000.snvs:snvs-rtc-lp: rtc core: registered 20cc000.snvs:snvs-r as rtc0
i2c /dev entries driver
IR NEC protocol handler initialized
IR RC5(x/sz) protocol handler initialized
IR RC6 protocol handler initialized
IR JVC protocol handler initialized
IR Sony protocol handler initialized
IR SANYO protocol handler initialized
IR Sharp protocol handler initialized
IR MCE Keyboard/mouse protocol handler initialized
IR XMP protocol handler initialized
pxp-v4l2 pxp_v4l2: initialized
usbcore: registered new interface driver uvcvideo
USB Video Class driver (1.1.1)
imx2-wdt 20bc000.wdog: use WDOG_B to reboot.
imx2-wdt 20bc000.wdog: timeout 60 sec (nowayout=0)
Bluetooth: HCI UART driver ver 2.3
Bluetooth: HCI UART protocol H4 registered
Bluetooth: HCI UART protocol BCSP registered
Bluetooth: HCI UART protocol ATH3K registered
usbcore: registered new interface driver btusb
usbcore: registered new interface driver ath3k
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pltfm: SDHCI platform and OF driver helper
/soc/aips-bus@02100000/usdhc@02190000: voltage-ranges unspecified
sdhci-esdhc-imx 2190000.usdhc: could not get ultra high speed state, work on normal mode
sdhci-esdhc-imx 2190000.usdhc: Got CD GPIO
sdhci-esdhc-imx 2190000.usdhc: No vmmc regulator found
sdhci-esdhc-imx 2190000.usdhc: No vqmmc regulator found
mmc0: SDHCI controller on 2190000.usdhc [2190000.usdhc] using ADMA
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
usbcore: registered new interface driver snd-usb-audio
fsl-asrc 2034000.asrc: driver registered
NET: Registered protocol family 26
mmc0: host does not support reading read-only switch, assuming write-enable
usb 1-1: new high-speed USB device number 2 using ci_hdrc
NET: Registered protocol family 10
sit: IPv6 over IPv4 tunneling driver
mmc0: new high speed SDHC card at address aaaa
NET: Registered protocol family 17
mmcblk0: mmc0:aaaa SL16G 14.8 GiB
can: controller area network core (rev 20120528 abi 9)
NET: Registered protocol family 29
 mmcblk0: p1 p2
can: raw protocol (rev 20120528)
can: broadcast manager protocol (rev 20120528 t)
can: netlink gateway (rev 20130117) max_hops=1
Bluetooth: RFCOMM TTY layer initialized
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM ver 1.11
Bluetooth: BNEP (Ethernet Emulation) ver 1.3
Bluetooth: BNEP filters: protocol multicast
Bluetooth: BNEP socket layer initialized
Bluetooth: HIDP (Human Interface Emulation) ver 1.2
Bluetooth: HIDP socket layer initialized
8021q: 802.1Q VLAN Support v1.8
Key type dns_resolver registered
dhd_module_init in
snvs_rtc 20cc000.snvs:snvs-rtc-lp: setting system clock to 2019-05-31 05:57:49 UTC (1559282269)
hub 1-1:1.0: USB hub found
gpio_dvfs: disabling
VSD_3V3: disabling
hub 1-1:1.0: 2 ports detected
can-3v3: disabling
ALSA device list:
  No soundcards found.
EXT3-fs (mmcblk0p2): error: couldn't mount because of unsupported optional features (2c0)
EXT2-fs (mmcblk0p2): error: couldn't mount because of unsupported optional features (2c0)
EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
devtmpfs: mounted
Freeing unused kernel memory: 428K (80b39000 - 80ba4000)
EXT4-fs (mmcblk0p2): re-mounted. Opts: data=ordered
usb 1-1.1: new high-speed USB device number 3 using ci_hdrc
devpts: called with bogus options
random: dd urandom read with 9 bits of entropy available
fec 2188000.ethernet eth0: Freescale FEC PHY driver [Generic PHY] (mii_bus:phy_addr=2188000.ethernet:00, irq=-1)
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
fec 2188000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
random: nonblocking pool is initialized

PuTTY settings

enter image description here

Help:

We are unable to identify whether the system goes into sleep/suspend mode when it run continuously. Even if it enters such a state, should it not ask for a login and password when it is woke up through a terminal window?

We thought the problem could be with the more log messages getting printed on the console, so we printed minimum number of log messages to check if the system doesn't get blocked because of it. But the problem still persists.

We keep the UART pins of the console cable connected to the Tx/Rx pin of the board throughout the test run while the USB end is let loose and not connected to the desktop/laptop . Could that be a problem?

We are finding it difficult to narrow down the problem.

  • If you leave putty connected to the console all the time, do you still see the stoppage? – Mark Plotnick Jun 1 at 7:42
  • When the console is unresponsive, does typing Control-Q unfreeze it? – Mark Plotnick Jun 1 at 7:44
  • Yes. typing Ctrl+Q unfreezes the console and the application resumes its operation. Upto an hour we have kept putty connected and checked the log message, we did not observe any stoppage. The stoppage happens when we close the putty and run the device unmonitored for long periods of time. – rudhan-sacra Jun 1 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.