Artificial Intelligence and Machine Vision

Creating an optical computer  Note on creating an optical computer.  What is Optical Computer? A laptop is a microchip based computer and uses electricity and transisters to compute. An optical computer uses photons to compute.  How does it compare to a typical laptop? A modern desktop computer has about 5 TFLOPS (5 x 10^16 floating calculations per second). With an optical computer, there is no limit in the calcuations per second.   Is an optical computer faster than a quantuam computer?  In 2016, the fastest known quantum computer has 2000 qubits, which is 1000 faster than 512 qubits.  With an optical computer, there is no artificial limitation like 2000 or 500 qubits.   What's the theoretical compute limit on an optical computer?  There is a limit of speed of light. For now, the only artificial limitation is how we design the first prototype.  How much electricity energy does it require?  The first POC should use less than 1...

In this blog, I will describe how we reduced the noise of the Time-Of-Flight sensor in our AR glasses prototype. Types of noise - systematic noise    note: caused by imperfect sinusoidal modulation - random noise    note: by shot noise. use bilateral filtering Motion artifacts reduction note: when motion is observed on a target object, we have motion artifacts observed in the tof sensor.  This happens when TOF measurement is recorded sequentially.  And, this causes doppler effects. fix: - use Plus and Minus rules    -- reference:        1) "Time of flight motion compensation revisited"  (2014)        2) "Time of flight cameras: Principles, Methods and Applications" (2012) Physics-based MPI reduction fix: - use 2K+1 frequency measurements for K inferencing paths in absence of noise. Per-pixel temporal processing of raw ToF measurements fix: - matrix pencil method - Prong's met...

This blog is about how to train a neural network to extract depth maps from videos of moving people captured with a monocular camera. Note: With a monocular camera, extracting the depth map of moving people is difficult.  Difficulty is due to the motion blur and the rolling shutter of an image.  However, we can overcome these limitations by predicting the depth maps by the model trained with a generated dataset using SfM and MVS from the normalized videos. This normalized dataset can be the basis of the training set for the neural network to automatically extract the accurate depth maps from a typical video footage, without any further assistance from a MVS. To start this project with a SfM and a MVS, we will use TUM Dataset. So, the basic idea is to use SfM and Multiview Stereo to estimate depth, while serves as supervision during training. The RGB-D SLAM reference implementation from these papers are used: - RGB-D Slam (Robotics OS) - Real-time 3D Visual SLAM ...

Best Stereo SLAMs in 2017 are reviewed. Namely, (in arbitrary order) EKF-SLAM based,  Keyframe based,  Joint BA optimization based,  RSLAM,  S-PTAM,  LSD-SLAM,   Best RGB-D SLAMs in 2017 are also reviewed. KinectFusion,  Kintinuouns,  DVO-SLAM,  ElasticFusion,  RGB-D SLAM,   See my keypoints of the best Stereo SLAMs. Stereo SLAM Conditionally Independent Divide and Conquer EKF-SLAM [5]   operate in large environments than other approaches at that time uses both  close and far points far points whose depth cannot be reliably estimated due to little disparity in the stereo camera  uses an inverse depth parametrization [6] shows empirically points can be triangulated reliably, if their depth is less than about 40 times the stereo baseline.     - Keyframe-based  Stereo SLAM   - uses BA optimization in a local area to archive scalability.  ...

Training a DL model to find a local minima in n-dimensions can be a challenge.  Often, data scientists and ML engineers would use a gradient descent to optimize the path. Starting delta may be anywhere between 1e-3 or 1e-4.  Having a constant gridient would not fast-approach a local minima. There are few issues with this approach. 1) The first found local minima may not be the best minima.  It can be stuck in a sharp valley, where any deriviate change would raise the error rate above 50% or more. 2) The first found local minima may be a local mixima, as shown in the saddle point graph below. When optimizing on n-th dimensions of space of a DL model, the best approach is to find a flat valley, when the SGD can locate a stable ground and where error rates stay low or relatively small to what it landed in the best optimization. However, there are a better way than this. Instead of manually entering an initial gradient decent value and updating it...

ORB-SLAM - Uses   - Bundle Adjustment   - ORB features [9]   - A pose graph      - Essential graph      - a spanning tree        - loop closure links        - strong edges        - from covisibility graph   - covisibliity graph     - local covisible area     - tracking and mapping - mar point and keyframe selection   - generous spawning   - restrictive culling   - identify redundant keyframes   - improves robustness and lifelong operations - Stores map points:   - 3D position X(w,i) in the world coordinate system   - the viewing direction n(i)     - the mean unit vector of all its viewing directions     - the ray that joint the point with the optical center of the keyframe  - A representative ORB descriptor D(i)     - the associated ORB descriptor whose hamming di...

TUM RGB-D benchmark [38] - an excellent dataset to evaluate the accuracy of camera location - several sequences with accurate ground truth obtained with an external motion capture system KITTI - extracted 2000 corners - 512x384 - 752x480 - 1241x376 - 5 corners per cell Compute orientation & ORB descriptors  - novel, direct, semi-dense, LSD-SLAM [10]     - takes time to converge the depth values - PTAM benchmarks [4]   - manually selected two keyframes for initialization - align the keyframe trajectories using the similarity transformatione - scale is unknown - measure the absolute trajectory error (ATE) [38] - RGB-D SLAM [43]    - trajectories - use the similarity transform to check if the scale is well recovered. - align the trajectories with a rigid body transformation

In this blog, I will talk about how to create an AI to write programs for us.  This is a forward looking, exploratory note about how we should take the next step. Goal : 1) We want software that we will have to write only once. 2) This software will write all future applications. 3) When people want a new application, we simply collect the data for every specific task. 4) Then, through observation, this master program creates a new application for us. Current Computer Limitations - Our computer languages are created to support the CPU hardware structure and the memory structure in 1950s. - Our computer languages have evolved several times over.    note: they are a functional language, a structural language, object-oriented language and declarative language. -  They all require humans to write the code in logical and specific ways. Different Paradigm Current Situation: - Humans instructs a computer on rules of processing of input data. - Then, a com...

The traditional Automatic Speech Recognition (ASR) performs at about 85% accuracy rate.  At this rate, ASR users are often frustrated with the experience with using such a system. The tradition ASR is often fragile: 1) requires extensive modification of parameters, just to make it work. 2) requires extensive understanding of a language model and a acoustic model. 3) doesn't scale well to multiple languages. 4) hyper-sensitive to speaker variants. Deep Learning on the acoustic model has been introduced, but not much of gain in the accuracy. What if, we can do a DL from end to end? Connectionist Temporal Classification (2006) introduces an idea of using FFT on the frequency of a recording of a voice command and constructs a spectrogram at 8kHz.  At each spectrogram interval, a DL neural network can be assigned, individually. The basic idea is to have RNN output neurons to encode distribution over "symbols". The traditional ASR uses a phone...

A feature locator is essential in all CV domain.  It's the basis of the germetric transformation, epipolar geometry, to 3D mesh reconstruction. Many techniques - SIFT and other SLAM technologies, are available, but they require ideal environments to work in. To address the short comings: - sensitive to low texture environment - sensitive to low light envonrment - sensitive to high light environment (like outdoor day light with above 20k lux) - and many other issues I propose a CNN based neural network to detect 4 correspondences in an image A and an image B. Since it is tricky to have a neural network to predict a 4x4 affine matrix of rotation and translation, I separated the translation vector from the rotation vector. Basically, the ground truth data will be precalcalated with a generic SIFT with RANSAC to calculate the correspondences set P and P'. The L2 (Eucledean) distance will be used between a predicted value.  They are 4 points, so an averaged will ...

Pros - Lightweight - Full frame time-of-flight data (3D array) collected with a single laser pulse - Unambiguous direct calculation of range - Blur-free imager without motion distortion - Co-registeration of range and intensity for each pixel - Perfectly registered pixels within a frame - Ability to represent the camera-oblique objects - No precision scanning mechanism required - 3D flash LIDAR with 2D cameras (EO and IR) to combine 2D texture over 3D depth - Multiple 3D flash LIDAR cameras for full volumetric 3D scene - Lighter and smaller than point scanning systems - Non-moving parts - Lower power consumption - Ability to scan through range-gating, natural obscurants

1. Convert the disprepancy between each output and its target value into an error derivative. E = 1 / 2 Sigma (Tj - Yj)^2 j in Output   dE / dYj = - (Tj  -Yj)     2. Compute an error derivative in each hidden layer from error derivatives in the layer above. dE / dZj = dYj / dZj  * (dE / dYj) , where Zj is the sum of all outputs of i hidden units. , where Yi is the output of i hidden unit. , where Yj is the output of j unit dE / dZj =  Yj (1 - Yj) * (dE / dYj) , where Yj (1 - Yj) is dY / dZ of a nonlinear logic unit of y = 1 / (1 + e ^ -Z)  , where dY / dZ is y (1 - y) dE / dYi = Sigma(j) ( dZj / dYi ) * (dE / dZj) dE / dYi = Sigma(j) Wij * (dE / dZj) ,where dE / dZj is already computed in above layer. thus, dE / dWij = (dZj / dWij) * (dE / dZj) dE / dWij  = Yi * (dE / dZj)   Proof: y = 1 / (1 + e^-Z) = (1 + e^-Z)^-1 thus, dy/dz = -1 (-e^-z) / (1 + e^-z)^2  dy/dz = 1 / (1 + e^-z) * (e^-z / (1 + e^-z) ) = y ( 1 - y) because,  (e^-z) /...

Recall = TP / (TP + FN) Precision = TP / (TP + FP) SOLVE F1 Score = Recall * Precision /  (Recall + Precision) F1 = (TP / (TP + FN) * TP / (TP + FP))  / ( TP / (TP + FN) + TP / (TP +FP) ) F1 = ( TP^2 / (TP + FP) + (FP + FN)  )  / (TP (TP + FN) / (TP + FP) (TP + FN))  + ( TP (TP + FP) / (TP + FP) (TP + FN) ) F1 =  TP^2 / (2 TP^2 + FNTP + FPTP) F1 =  TP^2 / (TP^2 ( TP^2 + FN / TP + FP / TP)) F1 =  1 / (TP^2 + FN / TP + FP / TP) Thus,  F1 Score is a harmonic mean.   

Recover the camera matrices from estimated fundamental matrix F Use the camera matrices to compute the point correspondences. Without K1 and K2 (internal calibration), recover a projective transforme With K1 and K2, recover the metric reconstruction. The epipolar constraint defines a line, since x2^T F x1 = l1^T x1 = 0 An epipolar line corresponding to the point x2, because l1^T x1 = 0 Thus x2 is on the epipolar line A point e is epipole, when the epipolar lines meet at this point e. Must satisfy: F e1 = 0 or e2^T F = 0

In this blog, I will discuss how to calibrate a suite of sensors used in a L2 autonomous prototype vehicle. Note: - To ensure a dataset generated from a L2 autonomous prototype, calibrate all sensors on board per each trip. In our autonomous vehicle prototypes, we use 6 - Cameras - we use 6 cameras - cropped native resolution from 1600x900 to smaller images - are in native BGR format. - with auto-exposure with a maximum limit of 20 ms. - use Bayer8 format for 1 byte per pixelin encoding - 1/1.8" CMOS sensor for 12 Hz capture frequency. - positions:   - one front center camera   - one front side mirrow camera per side   - one rear center camera   - one rear door centered camera per side 5 - Long Range RADARs - we use 5 sensors of RADAR - @ 13Hz capture frequency at 77 Ghz - measures distance & velocity, independently, in one cycle - positions:   - one front bumper center radar   - one front side mirror radar per side   - one...

Project Clear Sky Note: - This technology was created by me to enhance driving or flying conditions in poor weather. - This project is open sourced under GPL v3. - For the closed source for a commercial product, please contact dparksports at gmail dot com - Copyright 2010

Why RANSAC? Because we want a model of good feature matches. - Inliers:   - Good matches - Outlier:   - Bad matches Interest points (500/image) (640x480) What other algorithms are available? - Exhaustive search   - for each featfure, compare all other feactures in other images - Hashing   - compute a short descriptor from each feature vector - Nearest neighbor techniques:   - k-trees and their variants Putative correspondences (268) (Best match,SSD Outliers (117) (t=1.25 pixel; 43 iterations) How about outlier rejection? - use SSD (patch1, patch2) > threshold - 1-NN: SSD of the closest matches How to handle too many outliers? RANSAC loop: Select four feature pairs (at random) Compute homography H (exact) Compute inliers where  SSD(pi’, H pi) < ε Keep largest set of inliers Re-compute least-squares H estimate on all of the inliers Final inliers (262)

Before we talk about the projection matrix of the depth correspondces, we need to know two things: - Camera extrinsics - Camera intrinsics Camera extrinsics maps the world coorinates to the camera coordinates. For the simplicity of the camera, it is a pinhole camera without lenses.  I'll talk about the lenses, the focal length, the lense aberation, the pixel sensor dimension, etc in Camera intrincs. So, locating an object in two images and projecting in the camera space is not that straight. But, it will be a straight process with the application of Machine Learning. I'll talk about the next part of the series in applying the deep neural network to optimizing the homographic projection and have it robust in low texture settings including low light. Deep Neural Network - Estimating Homography to address: - low texture environment - outside light conditions ( gamma > 2kLs) - robust as or better than SfM or other SLAM techniquese First, we need to locate the ...

Direct Linear Transform (DLT) Using Direct Linear Transformation to find Homography. With the 4 correspondences, we can solve Direct Linear Transform (DLT) for the homography matrix H. However, a correspondence model may contain both inliers and outliers. Random Sample Consensus (RANSAC) In order to filter out the outliers, to reconstruct the model, use RANSAC in an iterative fashion. For example, for every set of 4 correspondences randomly picked, the occurrence of H is calculated. Using this H value, we can determine the outlier correspondences. With only inliers, we can recalculate H for each set of 4 correspondences.

How to Create a holographic display and camcorder In the last part of the series "How to Create a Holographic Display and Camcorder", I talked about what the interest points, descriptors, and features to find the same object in two photos. In this part of the series, I'll talk about how to extract the depth of the object in two photos by calculating the disparity between the photos. In order to that, we need to construct a triangle mesh between correspondences. To construct a mesh, we will use Delaunnay triagulation.  Delaunnay Triagulation - It minimizes angles of all triangles, while the sigma of triangles is maximized. The reason for the triangulation is to do a piece wise affine transformation for each triangle mapped from a projective plane A to a projective plane B. A projective plane A is of a camera projective view at time t, while a projective plane B is of a camera projective view at time t+1. (or, at t-1.  It really doesn't matter)...