Existing deep learning approaches have achieved high performance in encrypted network traffic analysis tasks. However, practical requirements such as open-set recognition on dynamically changing tasks (e.g., changes in the target website list), challenge existing methods. While few-shot learning and open-set recognition methods have been proposed for domains such as computer vision, few-shot open-set recognition for encrypted network traffic remains an unexplored area. This paper proposes a task adaptive siamese neural network for open-set recognition of encrypted network traffic with bidirectional dropout data augmentation. Our contributions are three-fold: First, we introduce generated positive and negative pairs into the siamese neural network training process to shape a more precise similarity boundary through bidirectional dropout data augmentation. Second, we utilize Dirichlet Process Gaussian Mixture Model (DPGMM) distribution to fit the similarity scores of the negative pairs constructed by the support set of each query task, and create a new open-set recognition metric. Third, by leveraging the extracted features at coarse and fine granular levels, we construct a hierarchical cross entropy loss to improve the confidence of the similarity score. Extensive experiments on a network traffic dataset and the Omniglot dataset demonstrate the superiority and generalizability of our proposed approach.