NVIDIA Jetson板子上安装nvidia docker需要注意的问题
一般来说，使用NVIDIA官网上下载的sdk image烧写入SD卡或者使用sdk manager/jet package连到板子安装全部软件后，里面什么都有了，包括docker和nvidia docker里面都有了，使用nvidia docker可以把GPU用起来，但是如果误删或者不小心弄坏了docker，或者为了减少发布系统的image的大小，尽量减少不需要安装的软件包，比如想从最基本的Basic Packages for L4T 开始根据需要一点一点的添加，那么自己安装nvidia docker甚至包括docker可能是需要的。
使用用Basic Packages for L4T制作的image烧写SD卡后，里面只有docker但是没有nvidia-docker2或者nvidia-container-toolkit，从R32.4.3开始，里面默认提供的docker是比较新一点的19.03，docker19.03是安装nvidia-container-toolkit(之前的老版本叫nvidia-docker2，已经deprecated了)的基础，有了它就可以安装新版的nvidia-container-toolkit了(当然，非要保持安装nvidia-docker2也是依然可以的)。
对于R32.4.2或之前的版本，使用的docker18，只能在上面安装nvidia-docker2，想安装最新的比较好用一点的nvidia-container-toolkit的话，可以把docker卸载掉再安装docker19.03，Jetson是arm64芯片，安装docker的步骤参见https://docs.docker.com/install/linux/docker-ce/ubuntu/，注意在第3步里选择arm64版的命令就行：
安装nvidia-container-toolkit或者nvidia-docker2，直接使用apt-get install命令就行，因为/etc/apt/sources.list.d/nvidia-l4t-apt-source.list这个源已保证可以找到你的板子对应的安装包，千万不要按照官网https://github.com/NVIDIA/nvidia-docker上面列的步骤去安装，那些是非Jetson板子的PC/Server上使用的，但是这网页上没有任何这方面的提示，这里就是个大坑很容易误导人，让人以为jetson序列板子上也是这么安装，我就被曾带坑里了，沿着那条路出错了拼命查错想解决问题，浪费了大半天时间，安装那些步骤安装是可以成功的，但是创建/启动docker容器时会报错:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request\\\\n\\\"\"": unknown.
ERRO[0000] error waiting for container: context canceled
于是使用下面的命令查找具体错误原因:
nvidia-container-cli -k -d /dev/tty info
发现是缺少so库:
I0821 03:01:49.655279 12951 nvc.c:282] initializing library context (version=1.2.0, build=d22237acaea94aa5ad5de70aac903534ed598819)
I0821 03:01:49.655588 12951 nvc.c:256] using root /
I0821 03:01:49.655646 12951 nvc.c:257] using ldcache /etc/ld.so.cache
I0821 03:01:49.655683 12951 nvc.c:258] using unprivileged user 65534:65534
I0821 03:01:49.655798 12951 nvc.c:299] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0821 03:01:49.656379 12951 nvc.c:301] dxcore initialization failed, continuing assuming a non-WSL environment
W0821 03:01:49.656998 12951 nvc.c:172] failed to detect NVIDIA devices
I0821 03:01:49.657613 12952 nvc.c:192] loading kernel module nvidia
E0821 03:01:49.658462 12952 nvc.c:194] could not load kernel module nvidia
I0821 03:01:49.658521 12952 nvc.c:204] loading kernel module nvidia_uvm
E0821 03:01:49.659007 12952 nvc.c:206] could not load kernel module nvidia_uvm
I0821 03:01:49.659062 12952 nvc.c:212] loading kernel module nvidia_modeset
E0821 03:01:49.659610 12952 nvc.c:214] could not load kernel module nvidia_modeset
I0821 03:01:49.660696 12953 driver.c:101] starting driver service
E0821 03:01:49.662765 12953 driver.c:161] could not start driver service: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory
I0821 03:01:49.664031 12951 driver.c:196] driver service terminated successfully
nvidia-container-cli: initialization error: driver error: failed to process request
感觉奇怪，怎么可能缺少文件呢，开始总想着怎么找个文件拷贝进去补上，可是官网上找不到说明，后来在国外一网页上看到有人说libnvidia-ml.so是NVML库，也就是管理方面的库文件，和nvidia-smi一样，jetson平台上是没有的，原来如此，那就意味着按照https://github.com/NVIDIA/nvidia-docker上面列的步骤去安装的nvidia-container-toolkit肯定是不适用于Jetson板子上的，可是NVIDIA官网上又找不到任何关于jetson板子上的nvidia-container-toolkit的安装包，那就只好试试nvidia-l4t-apt-source本身有没有提供安装源吧，于是先彻底删除误装的非Jetson平台版的nvidia-docker2或nvidia-container-toolkit了，需要purge确保删除了所有nvidia container相关的包:
apt-get remove nvidia-container-toolkit
dpkg --purge nvidia-docker2
dkpg --purge libnvidia-container-tools
dkpg --purge libnvidia-container0:arm64
dkpg --purge libnvidia-container1:arm64
然后直接执行:
sudo apt-get update && sudo apt-get install nvidia-container-toolkit
发现还真可以安装，nvidia-docker2也是可以这样直接安装，绕了一大圈跌进大坑里，爬上来发现原来这么简单啊，浪费大把时间。装完nvidia-container-toolkit执行 dpkg -l|grep nvidia 可以看到nvidia-container相关的包:
ii libnvidia-container-tools 0.9.0~beta.1 arm64 NVIDIA container runtime library (command-line tools)
ii libnvidia-container0:arm64 0.9.0~beta.1 arm64 NVIDIA container runtime library
ii nvidia-container-toolkit 1.0.1-1 arm64 NVIDIA container runtime hook
docker有了，nvidia-container-toolkit也有了，那可以用了吧，没想到后面还有一个大坑，使用 docker run --gpus all ...这样的命令时可以创建镜像的容器了，可是进入容器后，发现我们基于deepstream和一些CV方面的AI模型开发的视频识别程序跑不起来，之前在在host上可以跑得很好，用官网下载的sdk image烧写的SD卡里也没问题，为何用Basic Packages for L4T制作的image烧写SD卡后安装了docker和nvidia docker后就不行呢? 检查发现cuda没有从host上挂载到容器内，一些deepstream和AI模型所需的库，主要是tensorrt那些so库，都在容器内找不着，可是在官方的sdk image烧写的系统里启动同样镜像的容器内运行就没这些问题，搞不懂哪里出了错，浪费了差不多两天时间反复试验，NVIDIA官网上是找不到任何线索的，最后反复比较和官方image的差异发现nvidia docker方面，官方的完整镜像的系统起来后看到它就是多了这几个包：
于是同样试着直接安装这些包(visionworks我们暂时没用到):
sudo apt-get install nvidia-container-csv-cuda
sudo apt-get install nvidia-container-csv-cudnn
sudo apt-get install nvidia-container-csv-tensorrt
然后，再创建容器进入容器，一检查发现cuda和/usr/lib/aarch64-linux-gnu/下那些host同样路径下具有的so库也全都有了，这表明上面的csv包起作用了，host上的相关文件都挂载进来了，然后启动我们的视频识别系统，可以正常运行了！
host上的哪些文件映射挂载到容器内是通过这几个配置文件控制的：
/etc/nvidia-container-runtime/host-files-for-container.d/l4t.csv
/etc/nvidia-container-runtime/host-files-for-container.d/cuda.csv
/etc/nvidia-container-runtime/host-files-for-container.d/cudnn.csv
/etc/nvidia-container-runtime/host-files-for-container.d/tensorrt.csv
/etc/nvidia-container-runtime/host-files-for-container.d/visionworks.csv
总结：将sd卡烧写最基本最简单的l4t-base image后，并且安装了nvidia-docker2或者nvidia-container-toolkit后，nvidia docker可以跑起来，但是没有挂载host的cuda和deepstream，tensorrt的文件，这个原因就是上面的csv在安装nvidia-docker2或者nvidia-container-toolkit时是没有的，需要自己手工安装，这个对于JP安装的或者l4t完整版image烧写的卡，里面是已经自动安装了的，这就是为何烧写了l4t完整版image的Jetson Nano之类的板子上可以正常跑起来，但是烧写l4t-base image的Nano上即使安装了nvidia docker后还是不行，那些CSV在nvidia网站上没看到提及到，但是我们从最基本的l4t-base开始安装nvidia docker时需要安装这几个CSV包。
附:
Jetson板子上安装了正确版本的nvidia-container-toolkit后，执行nvidia-container-cli -k -d /dev/tty info 可以看到很多在Jetson板子上相对于非Jetson平台没有的文件是missing的，但是不会认为是错误:
-- WARNING, the following logs are for debugging purposes only --
I0821 03:20:19.523262 14989 nvc.c:281] initializing library context (version=0.9.0+beta1, build=77c1cbc2f6595c59beda3699ebb9d49a0a8af426)
I0821 03:20:19.523517 14989 nvc.c:255] using root /
I0821 03:20:19.523565 14989 nvc.c:256] using ldcache /etc/ld.so.cache
I0821 03:20:19.523596 14989 nvc.c:257] using unprivileged user 65534:65534
I0821 03:20:19.524223 14990 driver.c:134] starting driver service
I0821 03:20:19.537689 14989 nvc_info.c:585] requesting driver information with ''
I0821 03:20:19.544428 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-tls.so.32.4.3
I0821 03:20:19.545362 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-ptxjitcompiler.so.32.4.3
I0821 03:20:19.546814 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-glsi.so.32.4.3
I0821 03:20:19.548928 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-glcore.so.32.4.3
I0821 03:20:19.549514 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.32.4.3
I0821 03:20:19.551677 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-eglcore.so.32.4.3
I0821 03:20:19.556403 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1
I0821 03:20:19.558526 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libGLX_nvidia.so.0
I0821 03:20:19.558819 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libGLESv2_nvidia.so.2
I0821 03:20:19.559168 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libGLESv1_CM_nvidia.so.1
I0821 03:20:19.559548 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libEGL_nvidia.so.0
W0821 03:20:19.559819 14989 nvc_info.c:305] missing library libnvidia-ml.so
W0821 03:20:19.559871 14989 nvc_info.c:305] missing library libnvidia-cfg.so
W0821 03:20:19.559923 14989 nvc_info.c:305] missing library libnvidia-opencl.so
W0821 03:20:19.559968 14989 nvc_info.c:305] missing library libnvidia-compiler.so
W0821 03:20:19.560041 14989 nvc_info.c:305] missing library libvdpau_nvidia.so
W0821 03:20:19.560113 14989 nvc_info.c:305] missing library libnvidia-encode.so
W0821 03:20:19.560158 14989 nvc_info.c:305] missing library libnvcuvid.so
W0821 03:20:19.560239 14989 nvc_info.c:305] missing library libnvidia-fbc.so
W0821 03:20:19.560299 14989 nvc_info.c:305] missing library libnvidia-ifr.so
W0821 03:20:19.560380 14989 nvc_info.c:309] missing compat32 library libnvidia-ml.so
W0821 03:20:19.560459 14989 nvc_info.c:309] missing compat32 library libnvidia-cfg.so
W0821 03:20:19.560532 14989 nvc_info.c:309] missing compat32 library libcuda.so
W0821 03:20:19.560594 14989 nvc_info.c:309] missing compat32 library libnvidia-opencl.so
W0821 03:20:19.560671 14989 nvc_info.c:309] missing compat32 library libnvidia-ptxjitcompiler.so
W0821 03:20:19.560718 14989 nvc_info.c:309] missing compat32 library libnvidia-fatbinaryloader.so
W0821 03:20:19.560790 14989 nvc_info.c:309] missing compat32 library libnvidia-compiler.so
W0821 03:20:19.560841 14989 nvc_info.c:309] missing compat32 library libvdpau_nvidia.so
W0821 03:20:19.560891 14989 nvc_info.c:309] missing compat32 library libnvidia-encode.so
W0821 03:20:19.560967 14989 nvc_info.c:309] missing compat32 library libnvcuvid.so
W0821 03:20:19.561022 14989 nvc_info.c:309] missing compat32 library libnvidia-eglcore.so
W0821 03:20:19.561091 14989 nvc_info.c:309] missing compat32 library libnvidia-glcore.so
W0821 03:20:19.561171 14989 nvc_info.c:309] missing compat32 library libnvidia-tls.so
W0821 03:20:19.561211 14989 nvc_info.c:309] missing compat32 library libnvidia-glsi.so
W0821 03:20:19.561287 14989 nvc_info.c:309] missing compat32 library libnvidia-fbc.so
W0821 03:20:19.561357 14989 nvc_info.c:309] missing compat32 library libnvidia-ifr.so
W0821 03:20:19.561402 14989 nvc_info.c:309] missing compat32 library libGLX_nvidia.so
W0821 03:20:19.561473 14989 nvc_info.c:309] missing compat32 library libEGL_nvidia.so
W0821 03:20:19.561514 14989 nvc_info.c:309] missing compat32 library libGLESv2_nvidia.so
W0821 03:20:19.561586 14989 nvc_info.c:309] missing compat32 library libGLESv1_CM_nvidia.so
W0821 03:20:19.564841 14989 nvc_info.c:331] missing binary nvidia-smi
W0821 03:20:19.564882 14989 nvc_info.c:331] missing binary nvidia-debugdump
W0821 03:20:19.564918 14989 nvc_info.c:331] missing binary nvidia-persistenced
W0821 03:20:19.564938 14989 nvc_info.c:331] missing binary nvidia-cuda-mps-control
W0821 03:20:19.564987 14989 nvc_info.c:331] missing binary nvidia-cuda-mps-server
W0821 03:20:19.565142 14989 nvc_info.c:280] missing ipc /var/run/nvidia-persistenced/socket
W0821 03:20:19.565241 14989 nvc_info.c:280] missing ipc /tmp/nvidia-mps
I0821 03:20:19.566493 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvv4l2.so
I0821 03:20:19.566632 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/desktop-shell.so
I0821 03:20:19.566755 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/drm-backend.so
I0821 03:20:19.566907 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/EGLWLInputEventExample
I0821 03:20:19.567052 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/EGLWLMockNavigation
I0821 03:20:19.567216 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/fullscreen-shell.so
I0821 03:20:19.567395 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/gl-renderer.so
I0821 03:20:19.567610 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/hmi-controller.so
I0821 03:20:19.567790 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/ivi-controller.so
I0821 03:20:19.567963 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/ivi-shell.so
I0821 03:20:19.568148 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/LayerManagerControl
I0821 03:20:19.568329 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libilmClient.so.2.2.0
I0821 03:20:19.568488 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libilmCommon.so.2.2.0
I0821 03:20:19.568651 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libilmControl.so.2.2.0
I0821 03:20:19.568818 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libilmInput.so.2.2.0
I0821 03:20:19.568976 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libweston-6.so.0
I0821 03:20:19.569206 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/libweston-desktop-6.so.0
I0821 03:20:19.569382 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/simple-weston-client
I0821 03:20:19.569554 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/spring-tool
I0821 03:20:19.569735 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/wayland-backend.so
I0821 03:20:19.569916 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston
I0821 03:20:19.570095 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-calibrator
I0821 03:20:19.570324 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-clickdot
I0821 03:20:19.570426 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-cliptest
I0821 03:20:19.570525 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-content-protection
I0821 03:20:19.570618 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-debug
I0821 03:20:19.570733 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-desktop-shell
I0821 03:20:19.570846 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-dnd
I0821 03:20:19.570968 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-eventdemo
I0821 03:20:19.571156 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-flower
I0821 03:20:19.571281 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-fullscreen
I0821 03:20:19.571401 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-image
I0821 03:20:19.571529 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-info
I0821 03:20:19.571641 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-keyboard
I0821 03:20:19.571754 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-launch
I0821 03:20:19.571882 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-multi-resource
I0821 03:20:19.572013 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-resizor
I0821 03:20:19.572131 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-scaler
I0821 03:20:19.572249 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-screenshooter
I0821 03:20:19.572388 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-simple-dmabuf-egldevice
I0821 03:20:19.572524 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-simple-egl
I0821 03:20:19.572641 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-simple-shm
I0821 03:20:19.572767 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-simple-touch
I0821 03:20:19.572884 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-smoke
I0821 03:20:19.573004 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-stacking
I0821 03:20:19.573137 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-subsurfaces
I0821 03:20:19.573259 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-terminal
I0821 03:20:19.573379 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/weston/weston-transformed
I0821 03:20:19.573504 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so
I0821 03:20:19.573616 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvcompositor.so
I0821 03:20:19.573726 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvdrmvideosink.so
I0821 03:20:19.573843 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglglessink.so
I0821 03:20:19.573961 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglstreamsrc.so
I0821 03:20:19.574070 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvegltransform.so
I0821 03:20:19.574180 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvivafilter.so
I0821 03:20:19.574307 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvjpeg.so
I0821 03:20:19.574411 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvtee.so
I0821 03:20:19.574511 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvv4l2camerasrc.so
I0821 03:20:19.574612 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so
I0821 03:20:19.574714 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so
I0821 03:20:19.574877 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideocuda.so
I0821 03:20:19.574993 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosink.so
I0821 03:20:19.575120 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosinks.so
I0821 03:20:19.575234 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstomx.so
I0821 03:20:19.575335 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/libgstnvegl-1.0.so.0
I0821 03:20:19.575452 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/libgstnvexifmeta.so
I0821 03:20:19.575549 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/libgstnvivameta.so
I0821 03:20:19.575702 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/libnvsample_cudaprocess.so
I0821 03:20:19.575812 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/ld.so.conf
I0821 03:20:19.575924 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libEGL_nvidia.so.0
I0821 03:20:19.576039 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libGLESv1_CM_nvidia.so.1
I0821 03:20:19.576166 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libGLESv2_nvidia.so.2
I0821 03:20:19.576276 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/nvidia.json
I0821 03:20:19.576386 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1
I0821 03:20:19.576505 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libdrm.so.2
I0821 03:20:19.576628 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libGLX_nvidia.so.0
I0821 03:20:19.576741 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvapputil.so
I0821 03:20:19.576852 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so
I0821 03:20:19.576966 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketclient.so
I0821 03:20:19.577091 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketserver.so
I0821 03:20:19.577197 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvavp.so
I0821 03:20:19.577304 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0
I0821 03:20:19.577413 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so.1.0.0
I0821 03:20:19.577527 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so.1.0.0
I0821 03:20:19.577689 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0
I0821 03:20:19.577821 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcameratools.so
I0821 03:20:19.577937 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcamerautils.so
I0821 03:20:19.578064 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcam_imageencoder.so
I0821 03:20:19.578176 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcamlog.so
I0821 03:20:19.578306 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcamv4l2.so
I0821 03:20:19.578431 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvcolorutil.so
I0821 03:20:19.578539 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so
I0821 03:20:19.578648 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so
I0821 03:20:19.578792 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so
I0821 03:20:19.578906 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so.1.0.0
I0821 03:20:19.579025 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnveglstream_camconsumer.so
I0821 03:20:19.579134 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnveglstreamproducer.so
I0821 03:20:19.579236 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnveventlib.so
I0821 03:20:19.579337 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvexif.so
I0821 03:20:19.579442 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvfnet.so
I0821 03:20:19.579553 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvfnetstoredefog.so
I0821 03:20:19.579677 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvfnetstorehdfx.so
I0821 03:20:19.579788 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgbm.so
I0821 03:20:19.579894 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_boot.so
I0821 03:20:19.580001 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_camera.so
I0821 03:20:19.580102 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_force.so
I0821 03:20:19.580208 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_generic.so
I0821 03:20:19.580318 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_gpucompute.so
I0821 03:20:19.580422 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_graphics.so
I0821 03:20:19.580527 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_il.so
I0821 03:20:19.580630 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_spincircle.so
I0821 03:20:19.580734 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_tbc.so
I0821 03:20:19.580837 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvgov_ui.so
I0821 03:20:19.580949 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-eglcore.so.32.4.3
I0821 03:20:19.581076 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-egl-wayland.so
I0821 03:20:19.581188 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.32.4.3
I0821 03:20:19.581303 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-glcore.so.32.4.3
I0821 03:20:19.581456 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-glsi.so.32.4.3
I0821 03:20:19.581615 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-glvkspirv.so.32.4.3
I0821 03:20:19.581734 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-ptxjitcompiler.so.32.4.3
I0821 03:20:19.581873 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-rmapi-tegra.so.32.4.3
I0821 03:20:19.581996 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvidia-tls.so.32.4.3
I0821 03:20:19.582109 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so.1.0.0
I0821 03:20:19.582270 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so
I0821 03:20:19.582381 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
I0821 03:20:19.582490 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvll.so
I0821 03:20:19.582640 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so
I0821 03:20:19.582755 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmm_contentpipe.so
I0821 03:20:19.582866 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_image.so
I0821 03:20:19.582972 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite.so
I0821 03:20:19.583078 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_utils.so
I0821 03:20:19.583197 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so
I0821 03:20:19.583311 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmm_parser.so
I0821 03:20:19.583419 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmm.so
I0821 03:20:19.583547 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvmm_utils.so
I0821 03:20:19.583714 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvodm_imager.so
I0821 03:20:19.583837 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvofsdk.so
I0821 03:20:19.583949 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvomxilclient.so
I0821 03:20:19.584076 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvomx.so
I0821 03:20:19.584186 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvosd.so
I0821 03:20:19.584297 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
I0821 03:20:19.584410 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so
I0821 03:20:19.584545 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvphsd.so
I0821 03:20:19.584663 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvphs.so
I0821 03:20:19.584783 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so
I0821 03:20:19.584897 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so
I0821 03:20:19.585021 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so
I0821 03:20:19.585137 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
I0821 03:20:19.585264 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvtestresults.so
I0821 03:20:19.585370 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvtnr.so
I0821 03:20:19.585473 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvtracebuf.so
I0821 03:20:19.585594 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so
I0821 03:20:19.585714 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvv4l2.so
I0821 03:20:19.585829 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvv4lconvert.so
I0821 03:20:19.585938 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvvulkan-producer.so
I0821 03:20:19.586059 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libnvwinsys.so
I0821 03:20:19.586170 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libsensors.hal-client.nvs.so
I0821 03:20:19.586301 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libsensors_hal.nvs.so
I0821 03:20:19.586466 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libsensors.l4t.no_fusion.nvs.so
I0821 03:20:19.586579 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so
I0821 03:20:19.586688 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvvidconv.so
I0821 03:20:19.586795 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvvideocodec.so
I0821 03:20:19.586914 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/libvulkan.so.1.2.132
I0821 03:20:19.587024 14989 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json
I0821 03:20:19.587129 14989 nvc_info.c:154] selecting /lib/firmware/tegra21x/nvhost_nvdec020_ns.fw
W0821 03:20:19.587216 14989 nvc_info.c:403] missing directory /lib/firmware/tegra19x
W0821 03:20:19.587273 14989 nvc_info.c:403] missing directory /lib/firmware/tegra18x
W0821 03:20:19.587660 14989 nvc_info.c:459] missing device /dev/nvhost-nvdec1
W0821 03:20:19.587809 14989 nvc_info.c:459] missing device /dev/nvhost-ctrl-nvdla0
W0821 03:20:19.587877 14989 nvc_info.c:459] missing device /dev/nvhost-ctrl-nvdla1
W0821 03:20:19.587931 14989 nvc_info.c:459] missing device /dev/nvhost-nvdla0
W0821 03:20:19.588008 14989 nvc_info.c:459] missing device /dev/nvhost-nvdla1
W0821 03:20:19.588074 14989 nvc_info.c:459] missing device /dev/nvidiactl
W0821 03:20:19.588308 14989 nvc_info.c:459] missing device /dev/nvhost-nvenc1
I0821 03:20:19.588561 14989 nvc_info.c:642] requesting device information with opts: ''
I0821 03:20:19.588966 14989 nvc_info.c:660] listing device (null) ((null) at (null))
I0821 03:20:19.589024 14989 nvc.c:314] shutting down library context
I0821 03:20:19.589141 14990 driver.c:191] terminating driver service
I0821 03:20:19.590115 14989 driver.c:231] driver service terminated successfully