When inserting the needle for venipuncture the exact angle of insertion depends on?

Technology (Singap World Sci). Author manuscript; available in PMC 2015 Jun 26.

Published in final edited form as:

PMCID: PMC4482475

NIHMSID: NIHMS702232

Abstract

Venipuncture is pivotal to a wide range of clinical interventions and is consequently the leading cause of medical injury in the U.S. Complications associated with venipuncture are exacerbated in difficult settings, where the rate of success depends heavily on the patient's physiology and the practitioner's experience. In this paper, we describe a device that improves the accuracy and safety of the procedure by autonomously establishing a peripheral line for blood draws and IV's. The device combines a near-infrared imaging system, computer vision software, and a robotically driven needle within a portable shell. The device operates by imaging and mapping in real-time the 3D spatial coordinates of subcutaneous veins in order to direct the needle into a designated vein. We demonstrate proof of concept by assessing imaging performance in humans and cannulation accuracy on an advanced phlebotomy training model.

INNOVATION

The problems associated with difficult venous access have, in recent years, driven the development of technologies to improve the accuracy of cannula placement. Commercial devices use ultrasound, visible-light (VIS) or near-infrared (NIR) imaging to enhance vessel contrast; however, these devices leave the needle insertion (the ultimate determinant of success) to human hands. Here we introduce a device that integrates NIR imaging with a portable robot to detect and cannulate veins automatically, and we provide early evidence of its potential to improve clinical efficacy. The device may be developed for a broad range of applications, including pediatric, geriatric, emergency, and military use. The technology furthermore represents a step in the miniaturization and automation of surgical robotic systems for routine clinical interventions.

INTRODUCTION

The first step in many clinical interventions is to establish access to the venous bloodstream. Carried out 1.4 billion times each year1,2, venipuncture is the most commonly performed invasive clinical routine in the United States. Traditionally guided by visual and tactile inspection, successful venipuncture requires training, experience, and skill to achieve high rates of first-stick accuracy. Difficulties associated with the procedure can be exacerbated in challenging settings, where the likelihood of success depends heavily on the patient’s physiology and the practitioner’s experience. Consequently, venipuncture has been reported as the leading cause of injury to both patients (1.2 to 2 million injuries per year3) and practitioners (400,000 to 1 million injuries per year4,5) in the U.S. Failure rates have been estimated to range from 20% to 33% overall6,7 and from 47% to 70% in especially difficult populations8,9. Repeated failure to start an intravenous line has been shown to significantly increase the chance of tissue damage and bloodborne disease transmission, and may necessitate alternative pathways of much greater cost and risk10,11. In total, difficult or failed venipuncture is estimated to cost the U.S. healthcare system $4.7 billion per year1.

The problems associated with venipuncture have driven the development of numerous commercial technologies to assist clinicians in finding veins. These imaging devices fall into one of three categories. The first category of devices uses ultrasound to detect venous structure12. Ultrasound imaging systems have become widely adopted for clinical use due to their flexibility and relatively low cost. Ultrasound waves can penetrate readily into human tissue, allowing both superficial and deep tissue structures to be visualized with high resolution. However, for the purpose of venipuncture, ultrasound imaging demands a fairly long learning curve, and in many cases a second operator is needed. Additionally, image artifacts and speckle may be problematic, for example in obese patients, where the acoustic signal may be heavily backscattered or attenuated due to adipose in the hypodermis13. The second category of assistive devices uses VIS light to image veins. Such devices include the Venoscope Vein Finder by Venoscope LLC14 and the Veinlite LEDX by Translite LLC15. Given the limited penetration of VIS light through skin, these devices are used more often for schlerotherapy than for venous access. The final category consists of NIR imagers, two of which have achieved market entry: the VeinViewer, developed by Christie Digital, Inc. and the AV300 System, developed by AccuVein Inc. These devices have been shown to improve the accuracy of venous access, particularly in difficult or pediatric patients16–20. However, there are several major restrictions with these devices that limit their clinical adoption. First, these devices provide no information about the depth of the vein — arguably the most difficult parameter for clinicians to gauge. Second, since current imaging systems do not directly assist with needle insertion, the success of the procedure remains dependent on the clinician’s ability to perform accurately cannulate the vein. For these reasons, imaging devices are most often used in second- and third-line rescue pathways21.

In addition to aiding in vein detection, there is much research into robotically-guided needle insertion22,23 for a broad range of applications, including prostatectomy24,25, orthopedic and neurosurgery26, endoscopy27, brachytherapy28, and animal research29. Ultrasound-guided robots have also been developed for various applications30,31. However, while robotic devices have been shown to improve the accuracy, treatment time, and outcome of various surgical operations32,33, none of the existing systems are conducive to venous cannula placement. Very recently, another group34 has worked on developing a bench top device for autonomous, robotic venous access that combines 2D NIR imaging, range-finding, and ultrasound. While the researchers have indicated that the system is able to select a suitable vein with 83% accuracy, no measure of success has been provided for needle insertion, and to date the work has yet to be published in a peer-reviewed scientific journal.

Despite significant advances in the fields of medical imaging and robotics, there is no published or commercially-available technology that enables automated venous access. The technology described in this paper has the potential to improve clinical efficacy and safety, increase workflow, and reduce costs due to consumables. The technology can furthermore be extended for arterial access in the future.

RESULTS

Device design, operational protocol, and usability

To guide device development, we have established six design criteria (Table 1) based on clinical feedback and on-site observations of eight hospitals and three diagnostic centers; these specifications motivate the design choices described below. The device (Figs. 1a, 1b, Supplementary Video 1) is portable, self-contained, and can be used with minimal instruction. At the beginning of the procedure, the patient places his or her arm in a padded arm sleeve, and the device scans the arm to search for veins. After a suitable vein is selected, the robot positions the needle above the vein, confirms that the arm is stable, and inserts the needle. The device then disengages the needle, and the clinician performs the remainder of the protocol (for example, blood can be drawn through the catheter tube into collection vials). A mobile touchscreen interface (Samsung Series 7 Tablet) allows the clinician to visualize the procedure and intervene when needed. The interface also allows for intervention in cases of failure or emergency. The arm sleeve can be tightened to secure the arm, and includes a strap at the proximal end that can be used as a tourniquet (Supplementary Video 2). A disposable paper sheet is placed over the padded sleeve and serves as a sterile barrier between the device and the patient’s skin.

System design and architecture. (a) Functional prototype. The device measures 18 in3, weighs 13 kg, and features a detachable unit for sterilization, a removable blood tube holder, spring-loaded safety needles, a touchscreen interface, and a rechargeable battery. (b) Major functional components. Top left: 5 DOF mechanism (x, y, θ, φ, inj); Top right: imaging system; Bottom left: injection arm; Bottom right: full assembly. Labels: (i) patient arm constraint; (ii) stereo cameras; (iii) motorized camera positioning and calibration; (iv) motor encoder feedback; (v) needle with automated detachment. (c) Device data flow. Digital stereo image information is passed to a graphics processing unit for analysis, and then to the control system that guides the robot. User input/output is provided through a touchscreen interface. (d) Hardware architecture grouped by function. The mainboard (white) directs communication between the stereo imaging system (light blue), GPU (dark blue), motors and controllers (red), power system (gray), and interface (tan).

Table 1

Design criteria for prototype development.

The core components of the system include the stereo imager, the graphics processing unit (GPU), the robot mechanism, and the main control system (Fig. 1c). The hardware architecture is shown in Fig. 1d; the main control system directs communication between the imaging, analysis, and robotic components of the device. The device is powered by a 7 hr rechargeable battery (Tenergy NiMH 12V).

Near infrared imaging, image analysis, and robotic controls

The signal-to-noise ratio in vein imaging depends on the extent of hemoglobin absorption relative to melanin absorption and bulk tissue scatter. In comparison to VIS light (of which penetration depth is less than 1 mm), light in the NIR spectrum is relatively resilient to melanin absorption and tissue scatter (Fig. 2a) and thus can penetrate up to 4 mm through skin35. The device takes advantage of the increased penetration to improve the contrast of subcutaneous veins relative to background tissue. 12 arrays of light-emitting diodes (LEDs) provide reflectance illumination, and two CMOS cameras with increased NIR sensitivity detect the reflected light and capture images in real time (Fig. 2b). A number of standard optical techniques are applied (Fig. 2c): firstly, a 940 nm band-pass filter is positioned in front of each camera to eliminate ambient light; secondly, holographic diffusion filters are placed over the LEDs to increase optical isotropy and to allow forearm backscattering to be modeled as a Lambertian phenomenon; and thirdly, NIR polarizers are positioned orthogonally over the camera and LEDs to reduce specular reflection from superficial skin layers.

Real time vision-based robot guidance. (a) Optical properties in the NIR wavelength range. Absorption, scatter, and reflectance describe spectral characteristics of human skin. CCD sensitivity describes the quantum efficiency of the CMOS camera. Contrast is defined as the vein-to-background intensity ratio, IC = |IV − IB| / (IV + IB). DHb, deoxyhemoglobin; OHb, oxyhemoglobin. (b) Overview of imaging, vision, and control systems. The device operates by imaging and mapping the 3D spatial coordinates of subcutaneous veins in order to direct the needle into a designated vein without human intervention. (c) Multispectral stereo image acquisition under VIS and NIR light. (d) Image processing and vein segmentation. (e) 3D reconstruction of subcutaneous forearm veins. (f) Cannulation site selection, object tracking, and closed-loop kinematic controls.

The image analysis algorithms developed for the device extract the 3D structure of peripheral veins based on their geometric characteristics (Fig. 2d), which can be found in the Hessian matrix H. Veins, being tubular, exhibit negative curvature in one direction and near-zero curvature in the orthogonal direction. The geometry of an ideal dark tubular structure in a 2D image can be summarized by a simple set of descriptors: λ1 ~ 0, |λ1| << |λ2|, λ2 < 0, where λ1 and λ2 are the eigenvalues of H. Vessel enhancement has been previously demonstrated using the above descriptors36. Local normalization and thresholding is applied to segment the veins, after which the medial axis skeletons are extracted (Supplementary Fig. 1a).

Needle segmentation and tracking is performed by applying the Hough transform for line detection directly to the grayscale image37 (Supplementary Fig. 1b). The routine is implemented in real time by incorporating a hierarchical approach to the parameter search38. The needle tip is extracted using the Harris corner detector39, and the tip position is checked based on the known needle length.

The device employs principles of stereovision to achieve a depth resolution of 0.1–0.2 mm within the focal range. Stereo image pairs from two aligned cameras are analyzed to reconstruct the 3D scene. During device initialization, the camera intrinsic and extrinsic parameters are automatically calibrated based on a reference grid built into the device. Rectified images from the camera pair are then matched by a stereo correspondence algorithm that operates in real time on the GPU40; the method applies hierarchical belief propagation with constant memory costs and runtimes linear with image size41. A 3D point cloud model of the veins and arm surface is constructed from the depth map (Fig. 2e). A suitable site of cannulation is selected based on an aggregate score of image features (see Materials and Methods). The veins and needle are then tracked (Fig. 2f, Supplementary Video 3) via optical flow42. Tracking of the selected cannulation position is further refined via normalized cross-correlation template matching with position constraints.

The robot couples a 2 DOF gantry with a 3 DOF injection arm. Once the target vein is chosen, the robot kinematics and needle pose are calculated (Supplementary Fig. 2) and the gantry positions the needle above the vein. The injection arm then orients the needle laterally along the direction of the vein, brings the needle close to the skin surface, and inserts the needle tip into the vein at a 30° angle. The step precision of each stepper actuator is ~0.05 um (0.3°).

Robust optimization of the near infrared imaging system

Previous studies have compared the visibility of peripheral veins under NIR and VIS light. Various designs of NIR imaging systems for vein detection have also been developed, some of which have achieved commercial adoption. However, to our knowledge, there has yet to be described a systematic evaluation of the parameters that may affect peripheral forearm vein imaging. We applied orthogonal experimental design43,44 to compare the imaging proficiency of the device under NIR and VIS conditions across 24 patients (Supplementary Fig. 3a). For each patient, the left and right forearm and antecubital fossa (ACF) was scanned under VIS and NIR light, and the NIR images were segmented (Fig. 3). Five device parameters (wavelength, the angle-of-incidence of the light source to the arm, the distance of the light source from the arm, power per LED, and the orientation of the cross-polarizers) and three demographic parameters (age, body mass index (BMI), and Fitzpatrick skin type (FST)) were assessed. Imaging performance was defined as the length of segmented veins divided by the length of the forearm. All five parameters were found to significantly affect performance, and the optimal imaging conditions were determined over the sample (Supplementary Fig. 3b). (See Supplementary Information – Statistical Analysis for a description of the statistical approach.)

Forearm vein imaging and segmentation. (a) Veins extracted from NIR images based on vessel geometry. (i) Images captured under 940 nm diffuse, polarized light; (ii) Difference of Gaussians enhances edges across multiple image scales; (iii) Spectral low-pass filter (spatial convolution) dissolves small-scale noise while preserving venous structures; (iv) Veins enhanced by analyzing their 2nd order local geometries from the eigenvalues λ1, λ2 (λ1 >= λ2) and eigenvectors e1, e2 of the image Hessian matrix H; veins, which exhibit negative curvature in one direction and near-zero curvature in the orthogonal direction, are characterized based on their proximity in feature space to the geometrical model λ1 ~ 0, | λ1| << | λ2|, λ2 < 0; (v) A linear anisotropic matched filter, applied along the direction of least curvature, differentially highlights long, continuous vessels; (vi) Image contrast is locally normalized at small spatial scales; (vii) The resulting image is thresholded using Otsu’s method, and morphological post-processing filters are applied to remove small-length artifacts; (viii) Final segmentation of peripheral forearm veins shown as an overlay against the original NIR image. (b) Forearm images of four patients from different demographic groups. Top row: VIS image; Center row: NIR image; Bottom row: vein segmentation of NIR images. From left to right: light skin (I), low BMI (19.1); light skin (II), normal BMI (23.4); dark skin (V), moderately high BMI (26.8); dark skin (IV), very high BMI (35.0).

Comparison of device imaging to the clinical standard

For most interventions, the major veins of the forearm and ACF (particularly the basilic, cephalic, and median cubital veins) serve as the preferred access points for blood draws and IV's. We compared the effectiveness of the device in detecting these three veins to the effectiveness of the standard landmark technique performed by a certified phlebotomist. 101 patients spanning a broad range of demographics were recruited (Supplementary Fig. 4). The left and right forearm and ACF of each patient was imaged by the device using VIS and NIR light and then analyzed manually by the phlebotomist. The path length of detected (segmented) veins was quantified and normalized by forearm length. In the device’s case, path length was calculated directly from the image with a pixel to cm conversion. In the phlebotomist’s case, markers were drawn at 1 cm increments along the patient’s arm over detected veins, and the number of markers was counted. Age, sex, BMI, and FST, were documented for each patient and correlated in the analysis to device performance.

Across the study population, the path length of detected veins per patient was observed to correlate inversely with two demographic factors, BMI and FST, to significant effect (Fig. 4a). Neither age nor sex correlated significantly with path length, though in the case of age, a contributing factor may have been the lack of volunteers less than 10 years old. For all demographic groups, mean path length per patient was observed to be significantly greater with the device using NIR imaging compared to the phlebotomist. Furthermore, for all groups, the regression coefficients45 representing physiological influence on path length were significantly reduced in the device trials compared to the control trials, indicating that the imaging performance of the device is less affected by demographic variability.

Comparison of device imaging to the clinical standard. (a) Effects of three demographic groups — age, sex, body mass index (BMI), and Fitzpatrick skin type (FST) — on the total path length of detected veins, normalized by the forearm length. Path lengths were quantified during the imaging routine by the device under NIR light and by the clinician using standard visual and tactile techniques. Multiple regression of the device and clinical control data was performed independently using least-squares fit44. Significant effects were observed for BMI and FST, but not for age and sex. Analysis of covariance (ANCOVA)45 was conducted within each demographic grouping to determine whether the intercepts and slopes of the regression models were significantly different between the device and clinical controls. Significant differences (p-values < 0.05, indicated by asterisks (*)) between intercepts and slopes were observed in all cases. (b) Comparison of vein detection by the device and by the clinician. Results are categorized by into demographic groupings of age, sex, BMI, and FST. Numbers of samples per group are shown in parentheses. Statistical significance between the percent outcomes of the device and control trials was assessed using a one-sample t-test applied to percentages45. Asterisks indicate statistical significance between outcomes, defined as having two-tailed p-values less than 0.10 (*), 0.05 (**), or 0.01 (***). The device was observed to detect at least one, two, or three of the major peripheral forearm veins in a significantly greater percentage of the sample population compared to the clinical control.

We additionally determined the percentage of patients for which the major peripheral veins of the forearm and ACF were detected. The device was observed to detect at least one, two, or three of the major veins in a significantly greater percentage of the population compared to the phlebotomist (Fig. 4b). In 76 of 101 patients, at least one of the three major veins was detected by the phlebotomist, while the device detected at least one major vein in 99 of 101 patients (30.3% increase). All three major veins were detected by the phlebotomist in 51 of 101 patients, while the device detected all three veins in 81 patients (58.8% increase). Differences between phlebotomist and device outcomes were the most pronounced in more difficult patients, such as those presenting with dark skin or high BMI, indicating that the phlebotomist was affected by these conditions to a greater extent compared to the device.

Supplementary Fig. 5 compares vein segmentation and detection by the device under VIS and NIR imaging conditions, in relation to BMI and FST. A significant improvement in vein detection was observed using NIR imaging compared to VIS imaging.

We also evaluated the device's ability to autonomously select suitable locations for cannulation based on image features (See Materials and Methods for a description of the selection algorithm). In 89 of 101 patients, the automatically chosen cannulation site was confirmed by the phlebotomist as appropriate for venipuncture. The most frequently chosen vein was the median cubital vein in the ACF (successfully segmented in 92 patients and selected for cannulation in 70), followed by the cephalic vein (segmented in 75 patients and selected for cannulation in 17) and the basilic vein (segmented in 61 patients and selected in 8).

Device first-stick accuracy on phlebotomy training arm

We evaluated device accuracy and speed on a dark-skinned phlebotomy training model (NASCO LF01126U-Black). The model is well-known in the field of phlebotomy training and is classified as an "advanced" training arm. The model provides venous access for IV therapy and phlebotomy at all primary and secondary locations, including the ACF, forearm, wrist, and hand. The model comes with a palpatable skin layer with dark pigmentation, as well as an external fluid bag supplying artificial blood to the veins. Veins are positioned at varying diameters and depths beneath the skin surface to simulate different age groups and levels of difficulty.

As in the imaging studies, we employed orthogonal experimental design (Supplementary Fig. 6)43,44 to systematically determine the device settings that maximize cannulation accuracy. In 270 trials, the device was able to successfully cannulate the center of the target vein on the first try 100% of the time. On average, the mean distance of the needle tip from the vein center was 0.43 ± 0.21 mm. Mean distance was observed to change with different needle bevel orientations over all three vein diameters (Fig. 5a). Mean distance was observed to decrease as vein diameter decreased, possibly indicating that small veins are more likely to roll upon contact (Fig. 5b). Neither the angle nor speed of insertion was observed to significantly affect accuracy on the model. Needle diameter did not affect accuracy either, though veins smaller than 2.4 mm would necessitate higher gauge (lower diameter) needles. No needle deflection was observed in the study. The mean completion time per trial was 28.4 seconds (Fig. 5c). Time to completion did not vary significantly due to any of the test conditions. Individual time durations for the major steps in each cycle of the real-time process are shown in Fig. 5d. A demonstration of the device cannulation routine is shown in Supplementary Video 4.

Device cannulation accuracy on phlebotomy training model. (a) Cross-sectional representation of major veins of the model. Final needle tip positions within vessels are shown for each trial. Red: needle bevel oriented upward; Blue: needle bevel oriented downward. Significant differences between distribution means indicated by asterisks (two-tailed p-value < 0.05 (*)). (b) Device accuracy (measured by distances from vein centers) and robustness (measured by the signal-to-noise ratio (see Supplementary Information — Statistical Analysis) over different vein diameters. Box-plots show the median distances, as well as the upper quartiles, lower quartiles, and extremes. Circles show mean distances. Significant differences indicated by asterisks (two-tailed p-value < 0:05 (*), 0.01 (**)). Mean distance from the vein center was 0.43 +/− 0.21 mm. (c) Distribution of device completion times across all 270 trials. Mean completion time: 28.4 sec; min: 13.1 sec; max: 47.5 sec. (d) Duration of for one cycle from image acquisition to actuation is 0.1 sec (10 s−1 cycle rate). Box-plots show the duration for each step, as well as the upper and lower quartiles and the minimum and maximum durations.

DISCUSSION

This paper describes the development of a portable robotic device for automated venipuncture that has the potential to improve vein visualization, first-stick accuracy, and completion time independently of patient physiology and practitioner skill. The outcomes of the studies described here demonstrate 1) that the device is more readily able to detect peripheral forearm veins compared to one trained phlebotomist, especially in patients with comparatively difficult demographic profiles, and 2) that high cannulation accuracies and low completion times can be achieved on a phantom model.

In the comparative imaging studies, we allowed the phlebotomist to utilize visual and tactile techniques to find veins. In the majority of cases, the phlebotomist identified the median cubital vein in the ACF as the primary target for blood draws, and veins along the anterior lower forearm for IV infusions. Currently, the device suggests a cannulation target based on an aggregated score of various image features (see Materials and Methods) and displays the decision on the touchscreen interface in an overlay of the real time video. The clinician can either agree with the decision or select an alternative site of cannulation. Improving the decision model to incorporate higher level information, such as vein structure and anatomical prior knowledge, may allow the device to more closely replicate the deductive reasoning of the clinician.

NIR light, particularly at 940 nm, was observed to significantly improve the detection of peripheral upper limb veins compared to visible light. The improved detection at 940 nm is most likely due to the local DHb absorption peak as well as reduced scatter at longer wavelengths. We did not evaluate the system at wavelengths above 940 nm due the rapid decline in quantum efficiency for CMOS cameras past this range.

The cannulation studies on the advanced phlebotomy training arm provide proof-of-concept that a robotic device, particularly one guided by stereovision, can achieve the three-dimensional precision necessary to place a needle within small peripheral veins. The cannulation studies also demonstrate that the computational and robotic functions can all be achieved at a real-time rate. Finally, the studies provide evidence that a robot having the desired size, weight, speed, and cost is feasible.

Ongoing and future efforts will be directed toward further assessing the vein imaging and needle insertion functions of the device, with the goal being to demonstrate superiority over the current standard. Specifically, the device will be evaluated 1) in vitro on multiple forearm surrogates that accurately replicate the optical, mechanical, and physical properties of a pediatric human arm; 2) in vivo on IACUC approved rat tail blood draw protocols to observe device needle insertion, sample collection, time to completion, and potential safety risks; and finally 3) in humans through approved IRB protocols.

It is our contention that automated phlebotomy offers several key advantages over the current gold standard; these advantages include improving vein visualization for higher first-stick accuracy, mitigating clinical variability and human error, and increasing workflow. As an automated system, the device can furthermore enhance practitioner safety by minimizing the chance of needle-stick injury and associated blood-borne infections. The device, once translated, has the potential for impact in a number of arenas, including: 1) infant, elderly, and high BMI difficult venous access; 2) first-responder and emergency care; 3) military and trauma applications; and 4) hospital and medical facility phlebotomy. The device also has the potential to serve as a platform to merge automated phlebotomy with point of care diagnostics, which will allow critical patient information rapidly at the point of the blood draw. Finally, the underlying imaging, computer vision, pattern recognition, and robotic technologies developed for the device may be extended to applications such as arterial access, computer-aided diagnostics, and miniaturized robotic surgery.

MATERIALS AND METHODS

The device comprises several major components, shown in Supplementary Fig. 1 and described in individual sections below. These are: 1) a near-infrared optical imaging system for vein detection, 2) image analysis software for vein enhancement and segmentation; 3) 3D vein reconstruction software based on stereovision; 4) a vein selection algorithm; 5) robotic controls system and kinematics analysis; and 6) a 5 DOF robot that positions the needle. The main CPU is a Dell Precision M6400 laptop. The overall control software is written in LabView and directs communication between the imaging, analysis, and robotic components. The image acquisition hardware comprises two FireWire cameras and LEDs. The image analysis and computer vision software is run on a 1GB NVIDIA graphic processing unit (GPU) embedded within the laptop. The robot manipulator couples a 2 DOF gantry system with a 3 DOF arm that holds the needle. Each of these systems is described in detail below.

Near infrared imaging with diffuse, polarized light

The imaging system (Fig. 2b) consists of a NIR light source, a pair of stereo cameras, optical filters, a pair of polarizers, and an optical shell enclosing the device. The light source comprises arrays of 940 nm LEDs (Epitex SMC940). Two monochrome FireWire cameras with high sensitivity CCD sensors in the near-infrared range (Point Grey Firefly MV) are positioned above the needle. The field-of-view of the cameras allows the entire forearm and elbow to be viewed in one image.

Two layers of holographic diffusers (Edmund Optics) are used to improve the uniformity of intensity across the skin. 850–1060 nm bandpass filters (Edmund Optics) are used to mitigate ambient visible light while allowing the 940 nm NIR light. NIR polarizers (Edmund Optics) are placed in front of the LEDs and each camera. The polarizers filtering the light source is oriented orthogonally to the polarizers in front of the cameras. The polarized waves propagating to the arm surface and backscattering within the first few path lengths retain their polarization and are filtered upon reaching cameras. In contrast, waves propagating past the first few path lengths lose their initial polarity and are received by the cameras.

Multi-scale curvilinear vessel filter based on second-order geometry

Analyzing the 2nd-order information stored in the Hessian H has an intuitive justification for vessel detection36. Veins, being tubular, exhibit negative curvature in one direction and near-zero curvature in the orthogonal direction. Curvature and directionality are expressed by the eigenvalues λ1, λ2 (λ1 ≥ λ2) and eigenvectors e1, e2 of H. The local geometry of an ideal dark tubular structure in a 2D image can be summarized by the ratio RB of the eigenvalues at each pixel position:

|λ1|≈0|λ1|≪|λ2|RB=λ1λ2 λ2<0

A distinguishing property of background is that the magnitude of the derivatives (and thus the eigenvalues) is small. Thus we add the Frobenius norm of H to our feature descriptor:

Finally, we combine the two descriptors RB and S to compute an overall measure of vesselness V(σ) given weights β and c, as shown in Equation (1):

V(σ)={0, ifλ2<0exp(−Rβ22β2)(1−exp( −S22c2))}

(1)

Stereo calibration and 3D reconstruction

In the general case of reconstruction from multiple pinhole cameras viewing a 3D scene, the action of each camera is represented by a 3 × 4 projection matrix Pi containing the 3 × 3 matrix Ri describing camera orientation, a 3-vector ti giving camera position, and the camera intrinsic matrix A:

Then, given xij = PiXj relating the 3D points X = [X, Y, Z, 1]T to their 2D projections x = [x, y, 1]T, Pi is estimated numerically. We obtain a maximum likelihood estimation of Pi by minimizing the reprojection error R(ε), the sum-of-squares distance between measured and projected points. The epipolar geometry then summarizes the projective transformations between the stereo images x and x′:

with the fundamental matrix F relating epipolar constraints. A review of the calibration methods are presented by Zisserman et al.46 In our system, a circular calibration grid is presented at different depths from the camera. A precise motorized system was designed to move the cameras vertically to present the different depths. Solving the optimization problem described above allows the device to solve each camera's intrinsic and extrinsic parameters, and thus map the 2D image reference frame of each camera to the 3D camera frame.

3D representation of forearm veins are provided by determining the pixel disparity between the rectified images attained from the calibrated left and right cameras. We compute a disparity map for the full image by applying a real-time global stereo matching algorithm based on hierarchical constant space belief propagation. The disparities can then be used to arrive at a globally optimized 3D map of the veins.

Generally, the idea behind global stereo matching approach is to 1) formulate an energy function EX, containing a data term ED,X and a smoothness constraint term ES,X, that describes the strength of the match (Equation 2):

Ex(d)=ED,X(d)+ES,X(d)=ED,X(d)+∑Y∈N(X )MY,X(d)

(2)

2) define a 'jump cost' for violating the smoothness constraint, and 3) minimize the total energy. Global minimization is carried out in our case by loopy belief propagation of a message vector passed from each pixel to its four neighboring pixels. The value of d that minimizes EX at each pixel is selected for the final disparity map. To reduce computation time, a coarse-to-fine hierarchal approach is used. A detailed description of the hierarchical approach, and evaluation of its speed, is provided Yang et al.40 In our system, 320 × 240 image pairs are matched with 50 disparity levels at 20 Hz. The 3D volumes of filtered veins can then be reconstructed based on the stereo disparity map (Fig. 2d). The medial axis is extracted from the volume by computing the 3D flux skeleton45.

Selection of cannulation site

Given a vein v with skeleton S, we defined the suitability (ψp) of a 3D point p for puncture by observing the visual cues used by clinicians in their decision process. ψp is given in Equation (3).

ψp=[ω1LmaxvL+ω2ApmaxvAp+ω3Cmax vC+ω41−IpIv+ω51−2DL+ω6Vesselness∑ωi]

(3)

where L is the skeleton length of v; Ap is the vein cross sectional area about p; C the contrast; Ip the intensity at p; Ĭv the mean intensity of v; D the distance between p and the skeleton midpoint, and ωi, i = 1 … 6 the weights for each term, each set to 1 in our studies.

Robot mechanical design

The prototype has a total volume of 18 in3 and weighs 13 kg. The prototype couples a 2 DOF gantry system (Zaber Technologies) with a 3 DOF injection arm that servos the needle. The gantry allows translation in the x and y directions while the injection arm orients the needle with altitudinal and azimuth rotations and performs the actual insertion. The robot bears an “eye-in-hand” configuration for the needle and cameras: both the injection arm and the cameras are mounted such that the needle is always within the camera field-of-view. Total workspace is 10 × 6 in, sufficient to scan the length of the forearm and elbow. The step precision of the linear actuators is 0.05 µm and the step precision of the rotational actuators is 0.3 degrees, sufficient in allowing the system to a) exceed the stereo imaging resolution (0.1 mm) and b) achieve the desired injection precision (0.25 mm).

Closed-loop kinematic control of needle servoing

Needle steering is modulated by comparing the responses of an open-loop controller and a closed-loop controller. The open-loop controller measures the stepper motor response to calculate needle pose directly. The closed-loop controller determines the needle pose by relating the coordinate frames of the calibrated cameras to the needle frame and then to the fixed robot base frame, as is common in the art48.

At each cycle of the real-time loop, the needle pose computed by the closed-loop controller is compared to the open-loop response. Agreement to within 0.2 mm is considered acceptable, otherwise the action is flagged and the pose is reevaluated. A continuous flag period of t > 1 sec results in termination and a request by the device for recalibration. The mean cycle rate is ~10 Hz, depending primarily on the amount of translation at each cycle.

Safety design

Device safety designs are based on FDA guidelines for regulatory compliance of medical devices, wherein safety mechanisms are distributed across the device's sensor, effector, computer, and operator components. Electronic safety is monitored by a watchdog that activates fail-safe circuitry when a fault is detected. Mechanical safety is provided by ensuring that all degrees of freedom are locked when electrical torque is removed, in order to inhibit "free" robot motion. A detachable spring-locked needle is used which can be quickly disengaged from the robot. Finally, an intuitive user interface allows the clinician to operate the device with minimal training.

Blood collection

Blood collection tubes are loaded onto a housing unit prior to the phlebotomy. After the blood draw, the vials can be unloaded for analysis or storage, while the housing unit can be removed for sterilization.

Supplementary Material

Supplementary Video 1

Supplementary Video 2

Supplementary Video 3

01

Footnotes

AUTHOR CONTRIBUTIONS

Study concept and design, A.C., M.Y., T.M.; Optical system design and development, A.C.; Algorithms, software, and system control, A.C.; Data acquisition and processing, A.C.; Data analysis, A.C.; Drafting of manuscript, A.C., T.M.; Revisions and review of manuscript, M.Y.

COMPETING INTERESTS STATEMENT

All authors declare that they have no competing funding, employment, or personal financial interests.

REFERENCES

1. McCann M, Einarsdottir H, Van Waeleghem J-P, Murphy F, Sedgewick J. Vascular access management 1: An overview. J. Ren. Care. 2008;34:77–84. [PubMed] [Google Scholar]

2. Walsh G. Difficult peripheral venous access: Recognizing and managing the patient at risk. J. Assoc. Vasc. Access. 2008;13:198–203. [Google Scholar]

3. Kuensting LL, et al. Difficult venous access in children: Taking control. J. Emerg. Nurs. 2009;35:419–424. [PubMed] [Google Scholar]

4. Porta C, Handelman E, McGovern P. Needlestick injuries among health care workers. A literature review. AAOHN J. 1999;47:237–244. [PubMed] [Google Scholar]

5. Zaidi MA, Beshyah SA, Griffith R. Needle stick injuries: An overview of the size of the problem, prevention and management. Ibnosina J. Med. Biomed. Sci. 2010;2(2):53. [Google Scholar]

6. Rauch D, et al. Peripheral difficult venous access in children. Clin. Pediatr. 2009;48:895–901. [PubMed] [Google Scholar]

7. Jacobson AF, Winslow EH. Variables influencing intravenous catheter insertion difficulty and failure: An analysis of 339 intravenous catheter insertions. Heart Lung. 2005;34:345–359. [PubMed] [Google Scholar]

8. Lininger RA. Pediatric peripheral i.v. insertion success rates. Pediatr. Nurs. 2003;29:351–354. [PubMed] [Google Scholar]

9. Black KJL, Pusic MV, Harmidy D, McGillivray D. Pediatric intravenous insertion in the emergency department: Bevel up or bevel down? Pediatr. Emerg. Care. 2005;21:707–711. [PubMed] [Google Scholar]

10. Möller JC, Reiss I, Schaible T. Vascular access in neonates and infants — Indications, routes, techniques and devices, complications. Intensive Care World. 1995;12:48–53. [PubMed] [Google Scholar]

11. Campbell J. Intravenous cannulation: Potential complications. Prof. Nurse. 1997;12:S10–S13. [PubMed] [Google Scholar]

12. Maecken T, Grau T. Ultrasound imaging in vascular access. Crit. Care Med. 2007;35:S178–S185. [PubMed] [Google Scholar]

13. Fiegler W, Felix R, Langer M, Schultz E. Fat as a factor affecting resolution in diagnostic ultrasound: Possibilities for improving picture quality. Eur. J. Radiol. 1985;5:304–309. [PubMed] [Google Scholar]

14. Yucha CB, Russ P, Baker S. Detecting i.v. infiltrations using a Venoscope. J. Intraven. Nurs. 1997;20:50–55. [PubMed] [Google Scholar]

15. Katsogridakis YL, Seshadri R, Sullivan C, Waltzman ML. Veinlite transillumination in the pediatric emergency department: A therapeutic interventional trial. Pediatr. Emerg. Care. 2007;23:318–321. [PubMed] [Google Scholar]

16. Chapman LL, Sullivan B, Pacheco AL, Draleau CP, Becker BM. Vein-Viewer-assisted intravenous catheter placement in a pediatric emergency department. Acad. Emerg. Med. 2011;18:966–971. [PubMed] [Google Scholar]

17. Perry AM, Caviness AC, Hsu DC. Efficacy of a near-infrared light device in pediatric intravenous cannulation: A randomized controlled trial. Pediatr. Emerg. Care. 2011;27:5–10. [PubMed] [Google Scholar]

18. Kim MJ, et al. Efficacy of VeinViewer in pediatric peripheral intravenous access: A randomized controlled trial. Eur. J. Pediatr. 2012;171:1121–1125. [PubMed] [Google Scholar]

19. Sánchez-Morago G-VS, et al. Viewing veins with AccuVein AV300. Rev. Enferm. 2010;33:33–38. [PubMed] [Google Scholar]

20. Highton L, Ekwobi C. Use of the AccuVein device to map the superficial venous system. Eur. J. Plast. Surg. 2011;34:305–306. [Google Scholar]

21. Lavery I, Smith E. Peripheral vascular access devices: Risk prevention and management. Br. J. Nursing. 2007;16:1378–1380. 1382–1383. [PubMed] [Google Scholar]

22. Wiklund NP. Technology insight: Surgical robots — expensive toys or the future of urologic surgery? Nat. Clin. Pract. Urol. 2004;1:97–102. [PubMed] [Google Scholar]

23. Cleary K, Melzer A, Watson V, Kronreif G, Stoianovici D. Interventional robotic systems: Applications and technology state-of-the-art. Minim. Invasive Ther. Allied Technol. 2006;15:101–113. [PMC free article] [PubMed] [Google Scholar]

24. Sridhar AN, et al. Image-guided robotic interventions for prostate cancer. Nat. Rev. Urol. 2013;10:452–462. [PubMed] [Google Scholar]

25. Fütterer JJ, Barentsz JO. MRI-guided and robotic-assisted prostate biopsy. Curr. Opin. Urol. 2012;22:316–319. [PubMed] [Google Scholar]

26. Joskowicz L, et al. Image-guided system with miniature robot for precise positioning and targeting in keyhole neurosurgery. Comput. Aided Surg. 2006;11:181–193. [PubMed] [Google Scholar]

27. McGee MF, et al. A primer on natural orifice transluminal endoscopic surgery: Building a new paradigm. Surg. Innov. 2006;13:86–93. [PubMed] [Google Scholar]

28. Podder TK, et al. Reliability of EUCLIDIAN: An autonomous robotic system for image-guided prostate brachytherapy. Med. Phys. 2011;38:96–106. [PMC free article] [PubMed] [Google Scholar]

29. Kazanzides P, et al. Development of an image-guided robot for small animal research. Comput. Aided Surg. 2007;12:357–365. [PubMed] [Google Scholar]

30. Whitman J, Fronheiser MP, Smith SW. 3-D ultrasound guidance of surgical robotics using catheter transducers: Feasibility study. IEEE Trans. Ultrason. Ferroelectr. Freq. Control. 2008;55:1143–1145. [PubMed] [Google Scholar]

31. Abolmaesumi P, Salcudean SE, Zhu W-H, Sirouspour MR, DiMaio SP. Image-guided control of a robot for medical ultrasound. IEEE Trans. Robotics Automation. 2002;18:11–23. [Google Scholar]

32. Ahmed K, et al. Assessing the cost effectiveness of robotics in urological surgery — A systematic review. BJU Int. 2012;110:1544–1556. [PubMed] [Google Scholar]

33. Yu H-Y, Hevelone ND, Lipsitz SR, Kowalczyk KJ, Hu JC. Use, costs and comparative effectiveness of robotic assisted, laparoscopic and open urological surgery. J. Urol. 2012;187:1392–1399. [PubMed] [Google Scholar]

35. Bashkatov AN, Genina EA, Kochubey VI, Tuchin VV. Optical properties of human skin, subcutaneous and mucous tissues in the wavelength range from 400 to 2000 nm. J. Phys. D Appl. Phys. 2005;38:2543–2555. [Google Scholar]

36. Frangi AF, Niessen WJ, Vincken KL, Viergever MA. Multiscale vessel enhancement filtering. Med. Image Comput. 1998;1496:130–137. [Google Scholar]

37. Lo R, Tsai W-H. Gray-scale hough transform for thick line detection in grayscale images. Patt. Recog. 1995;28:647–661. [Google Scholar]

38. Qiu W, Zhou H, Ding MY, Zhang SG. New real time needle segmentation technique using grayscale Hough transformation. MIPPR 2007 Med. Imag. Parallel Process. Images Optim. Tech. 2007;6789:Q7890–Q7890. [Google Scholar]

39. Shi J, Tomasi C. Good features to track. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR94. 1994;94:593–600. [Google Scholar]

40. Jung J, Ho Y. CUDA-based GPU implementation of hierarchical belief propagation for fast stereo matching; Int. Conf. Embedd. Syst. Intell. Technol. (ICESIT); 2011. Feb, pp. 209–211. [Google Scholar]

41. Yang QYQ, Wang LWL, Ahuja N. A constant-space belief propagation algorithm for stereo matching; Comp. Vis. Patt. Recog. CVPR 2010 IEEE Conf; 2010. pp. 1458–1465. [Google Scholar]

42. Yilmaz A, Javed O, Shah M. Object tracking: A survey. ACM Computing Surveys. 2006;38:13. [Google Scholar]

43. Antony J, Kaye M. Methodology for Taguchi design of experiments for continuous quality improvement. Qual. World. 1995:98–102. [Google Scholar]

44. Bass I. Six sigma statistics with Excel and Minitab. Inform. Manage. 2007;41:374. [Google Scholar]

45. Ott L, Longnecker M. An Introduction to Statistical Methods and Data Analysis. 6th Edition. Cengage Learning, USA: 2010. [Google Scholar]

46. Hartley R, Zisserman A. Multiple View Geometry in Computer Vision. Vol. 2. Cambridge University Press, UK; 2004. p. 672. [Google Scholar]

47. Dimitrov P, Phillips C, Siddiqi K. Robust and efficient skeletal graphs; Proc. IEEE Conf. Comput. Vis. Patt. Recog; 2000. pp. 417–423. (CVPR 2000), Cat NoPR006621. [Google Scholar]

48. Zhao ZZZ, Liu YLY. Integrating camera calibration and hand-eye calibration into robot vision. 2008 7th World Congr. Intell. Control Automation. 2008:5721–5727. [Google Scholar]